Very Strange Network Problem

OK, I'm appealing to the greater powers than myself.

I have a network issue that I can't solve, and in fact, I don't even know what the problem is.


The only way I became aware of it was through my graphing of our spam.

At one point in time, we got a lot of spam. I implemented blackhole list checking, and a lot of it went away. I wanted something tangible to show people, so I graphed the spam loss in MRTG. Here is today's graph.



Yesterday is a normal graph. Steadily grows toward midnight, and drops off the next day. No worries. Today's is a bit different.

At 12:00, on the dot, we started receiving much less spam. The graph grows slower, which indicates a much lower flow. I saw this, and it made me curious. Usually SE Asian botnets don't just "disappear", but if there is a large quake that knocks out the power, it's been known to happen. I checked, and there's nothing above a 5.8 in the last day, so that isn't causing the problem.

I got more curious, and decided to check the total number of connections to our mail server, thinking maybe it was broke.

I ran this command line, which prints the number of connections in the left hand column, and the time of day (the hour) in the right. Here's what I got for total number of connections per hour:

root@twohearted:/var/log# cat maillog | grep "Feb  9"  | grep -v "9 0.:" | grep " connect" | awk '{print $3}' | awk -F: '{print $1}' | uniq -c

    699 10
    687 11
    334 12
    324 13
    316 14
    295 15
    347 16
    143 17

Alright, the number of mail connections get cut in half at noon. That's strange. I checked the number of spam catches we got:

root@twohearted:/var/log# cat maillog | grep "Feb  9"  | grep -v "9 0.:" | grep "cbl.abuseat.org" | awk '{print $3}' | awk -F: '{print $1}' | uniq -c
    188 10
    210 11
     40 12
     57 13
     30 14
     36 15
     58 16
     51 17

I'm mystified. The percentage of mail that is spam is ually around 1/2, as you can see in the first table. After the cut, the number of spam messages is 1/4, which is exactly the opposite of what you would expect if half of your email is legitimate, and came from people leaving early on Friday.

I began to wonder if there was something wrong at a lower level, so I wanted to check our DNS server, thinking maybe the mail server is broken or slow or something.

This is a table showing the number of MX lookups per hour. If our mail server was the problem, then there would be no difference at noon:

     41 05
    465 06
    526 07
    496 08
    498 09
    494 10
    522 11
    111 12
     81 13
     66 14
     65 15
     78 16
     24 17

1/5th the amount of MX requests after noon than before. Something much more serious is happening. I look at the number of requests total for the system.

root@smuttynose:/etc/tinydns/log/main# cat \@4000000045cc* current...
    181 05
   1863 06
   2028 07
   1968 08
   1972 09
   1866 10
   1948 11
    355 12
    386 13
    310 14
    296 15
    380 16
    118 17

And at this, I'm speechless. At noon, we lost almost all of our NS requests.

Does anyone have any idea of what this might be?