Unexplained linux server crash

games.ozfortress.com:30000

The server crashes at random times during play. Doesnt seem to occur with few players in the server, only when its full or spammy.

When the server crashes, the entire box goes down, and fails to respond for ~40 minutes. No mdmp file is created.

I just tried to run it with -debug, which may create a mdmp file.

Is anyone else experiencing such a thing? Is there anything I can do to debug furthur?

The server just came back after about an hour. The server has not rebooted, and during that time was unresponsive on all ports.

There is no mdmp file in the srcds installation, nor in /tmp.

also: srcds_i686 is still running.

Linux kernel:
Linux games 2.6.17-gentoo-r4 #9 SMP Thu Dec 28 09:17:03 EST 2006 i686 Intel(R) Xeon(TM) CPU 3.20GHz GenuineIntel GNU/Linux

Update:

Well, a bit more investigation (it appears that linux doesnt create mdmp files, it creates core files the server is definatly not crashing, nor is srcds.

It appears to be networking related, and because of that it seems like it cant possibly be srcds' (or FF's) fault.
Log from a crash:

 
L 09/17/2007 - 02:26:18: CMultiplayRules::DeathNotice
L 09/17/2007 - 02:26:18:  killer_ID: 9
L 09/17/2007 - 02:26:18:  killer_weapon_name: ff_weapon_supershotgun
L 09/17/2007 - 02:26:18:   begins with ff_, removing
L 09/17/2007 - 02:26:18:  userid (victim): 14
L 09/17/2007 - 02:26:18:  attacker: 9
L 09/17/2007 - 02:26:18:  weapon: weapon_supershotgun
L 09/17/2007 - 02:26:18: "Spk<9><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" killed "doggy<14><STEAM_0:X:XXXX><#FF_TEAM_RED>" with "weapon_supershotgun"
L 09/17/2007 - 02:26:22: CMultiplayRules::DeathNotice
L 09/17/2007 - 02:26:22:  killer_ID: 6
L 09/17/2007 - 02:26:22:  killer_weapon_name: ff_weapon_supershotgun
L 09/17/2007 - 02:26:22:   begins with ff_, removing
L 09/17/2007 - 02:26:22:  userid (victim): 16
L 09/17/2007 - 02:26:22:  attacker: 6
L 09/17/2007 - 02:26:22:  weapon: weapon_supershotgun
L 09/17/2007 - 02:26:22: "MickoLOL<6><STEAM_0:X:XXXX><#FF_TEAM_RED>" killed "Charlie<16><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" with "weapon_supershotgun"
L 09/17/2007 - 02:26:24: CMultiplayRules::DeathNotice
L 09/17/2007 - 02:26:24:  killer_ID: 17
L 09/17/2007 - 02:26:24:  killer_weapon_name: ff_weapon_supershotgun
L 09/17/2007 - 02:26:24:   begins with ff_, removing
L 09/17/2007 - 02:26:24:  userid (victim): 6
L 09/17/2007 - 02:26:24:  attacker: 17
L 09/17/2007 - 02:26:24:  weapon: weapon_supershotgun
L 09/17/2007 - 02:26:24: "Mang<17><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" killed "MickoLOL<6><STEAM_0:X:XXXX><#FF_TEAM_RED>" with "weapon_supershotgun"
L 09/17/2007 - 02:26:26: CMultiplayRules::DeathNotice
L 09/17/2007 - 02:26:26:  killer_ID: 10
L 09/17/2007 - 02:26:26:  killer_weapon_name: ff_weapon_supershotgun
L 09/17/2007 - 02:26:26:   begins with ff_, removing
L 09/17/2007 - 02:26:26:  userid (victim): 17
L 09/17/2007 - 02:26:26:  attacker: 10
L 09/17/2007 - 02:26:26:  weapon: weapon_supershotgun
L 09/17/2007 - 02:26:26: "rowlie<10><STEAM_0:X:XXXX><#FF_TEAM_RED>" killed "Mang<17><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" with "weapon_supershotgun"
L 09/17/2007 - 02:26:29: CMultiplayRules::DeathNotice
L 09/17/2007 - 02:26:29:  killer_ID: 17
L 09/17/2007 - 02:26:29:  killer_weapon_name: ff_grenade_normal
L 09/17/2007 - 02:26:29:   begins with ff_, removing
L 09/17/2007 - 02:26:29:  userid (victim): 10
L 09/17/2007 - 02:26:29:  attacker: 17
L 09/17/2007 - 02:26:29:  weapon: grenade_normal
L 09/17/2007 - 02:26:29: "Mang<17><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" killed "rowlie<10><STEAM_0:X:XXXX><#FF_TEAM_RED>" with "grenade_normal"
L 09/17/2007 - 02:27:31: "Spk<9><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" disconnected (reason "Spk timed out")
L 09/17/2007 - 02:27:31: "Obs<11><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" disconnected (reason "Obs timed out")
L 09/17/2007 - 02:27:31: "mrfluffy<12><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" disconnected (reason "mrfluffy timed out")
L 09/17/2007 - 02:27:31: "doggy<14><STEAM_0:X:XXXX><#FF_TEAM_RED>" disconnected (reason "doggy timed out")
L 09/17/2007 - 02:27:31: "Charlie<16><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" disconnected (reason "Charlie timed out")
L 09/17/2007 - 02:27:31: "merk<2><STEAM_0:X:XXXX><#FF_TEAM_RED>" disconnected (reason "merk timed out")
L 09/17/2007 - 02:27:31: "seaKing<8><STEAM_0:X:XXXX><#FF_TEAM_RED>" disconnected (reason "seaKing timed out")
L 09/17/2007 - 02:27:31: "wolfeh|RoK|<7><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" disconnected (reason "wolfeh|RoK| timed out")
L 09/17/2007 - 02:27:31: "rowlie<10><STEAM_0:X:XXXX><#FF_TEAM_RED>" disconnected (reason "rowlie timed out")
L 09/17/2007 - 02:27:31: "Seraphim<13><STEAM_0:X:XXXX><#FF_TEAM_RED>" disconnected (reason "Seraphim timed out")
L 09/17/2007 - 02:27:31: "newk<15><STEAM_0:X:XXXX><#FF_TEAM_RED>" disconnected (reason "newk timed out")
L 09/17/2007 - 02:27:31: "Mang<17><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" disconnected (reason "Mang timed out")
L 09/17/2007 - 02:27:31: "Kw<19><STEAM_0:X:XXXX><#FF_TEAM_RED>" disconnected (reason "Kw timed out")
L 09/17/2007 - 02:27:31: "Shpongle<3><STEAM_0:X:XXXX><#FF_TEAM_GREEN>" disconnected (reason "Shpongle timed out")
L 09/17/2007 - 02:27:31: "MickoLOL<6><STEAM_0:X:XXXX><#FF_TEAM_RED>" disconnected (reason "MickoLOL timed out")

And attached is a graph showing that it appears to be dropping out under load.

[Direhit]

If the NIC card is fine, you may want to look at the network settings in your kernel config or perhaps try different NIC drivers. Just out of curiosity, why are you using such an odd port number for the server?

Seemed like a good idea at the time. I'll probably move it to 27015 once our current TFC league finishes.

As for nic drivers and kernel configuration, it appears fine. Im waiting on the DC to provide a KVM to see if I can troubleshoot furthur during a crash.

Doing furthur debugging, it appears the FF has aggrevated a bug in the networking kernel driver I'm using.

Cant be 100% sure thats what it is, but there are a few occurances on the web of similar errors. My host has aparently done something with their switch too, as before I was only getting 10Mbps autodetect rather than the 100Mbps I was supposed to be getting.

/me crosses fingers that it wont crash again