Unplanned Server Outage - 21aug2014

[quadz]

Salutations,

Here's a synopsis of the unplanned server outage that occurred today.

At 1330 days of uptime, our primary server fragbait.tastyspleen.net crashed with a kernel panic in the Ethernet driver.

Subsequent attempts to reboot and/or power cycle the server were met with failures to bring up the network driver:

[    1.921773] e1000e: Intel(R) PRO/1000 Network Driver - 1.0.2-k2
[    1.921836] e1000e: Copyright (c) 1999-2008 Intel Corporation.
[    1.921945] e1000e 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[    1.922025] e1000e 0000:04:00.0: setting latency timer to 64
[    1.922140] e1000e 0000:04:00.0: irq 54 for MSI/MSI-X
[    2.172397] e1000e 0000:04:00.0: PCI INT A disabled
[    2.172467] e1000e: probe of 0000:04:00.0 failed with error -2
[    2.172552] e1000e 0000:04:00.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19
[    2.172629] e1000e 0000:04:00.1: setting latency timer to 64
[    2.172754] e1000e 0000:04:00.1: irq 54 for MSI/MSI-X
[    2.423008] e1000e 0000:04:00.1: PCI INT B disabled
[    2.423070] e1000e: probe of 0000:04:00.1 failed with error -2


After I opened a support ticket, I was able to watch the tech assigned by SoftLayer over the virtual KVM as he attempted to troubleshoot.

Sometimes when the tech would boot into a "rescue" kernel, the network driver would load. Sometimes after that when he booted back into the fragbait kernel the networking would still be OK--which is why we had a brief interval in the afternoon where the system was back up for awhile. However, on subsequent reboots the networking would go back to the error condition above.

The tech suggested updating the BIOS on the motherboard, but unfortunately this had no noticable effect on the problem.

The tech then continued running various diagnostic programs at occasional intervals, however the programs the tech was running depended on the Ethernet devices existing (but the devices did not exist, because the driver was unable to load, again related to the error above -- so these diagnostic tools were unable to run.)

So it was an interesting experience watching sometimes lengthy periods of inactivity punctuated by the running of another diagnostic tool which had no chance of working.

Eventually the problem was escalated to a higher tier tech who was aware of a rather bizarre problem with the particular motherboard on the fragbait server.

He was able to get the networking back up, and explained the issue as follows:

I am going to go ahead and put an account note on your account so that we can get this fixed in a faster fashion in the future.

There is a specific issue with the motherboard in this server. There is a trick of sorts that has to be performed as this motherboard can lose network access at times.

basically for the server fragbait2 which has the X7DBR-E we have to shutdown the machine, set the network interfaces to 0 then push the config to our switches.

Then wait a minute or two, then power back on the machine and then reset the speed of the interfaces back to the required speed which for this one is 1Gbps.

After that networking starts to work and is a documented issue for this board.

I did go ahead and add the notes for this server so hopefully any issues will be fixed in a quick manner.

So.... that's definitely an odd one. I'm glad somebody there knew a workaround. It will be interesting to see how long it lasts. (If it can last another 1330 days, I won't complain much. But who knows...)

[VaeVictis]

Never seen such an odd issue with a super micro board like that. Usually super micro boards are rock solid without any kind of issues

[fishxz]

sad to hear quadz my heart is bleeding for ur uptime

[bluemeanies]

1330

I think you misspelled 1337. Thanks for all your hard work and free service quadz.

And yes nerds...we've all heard the story of that company finding a server that had been up for 20 years dry walled in behind a wall in some basement. That uptime seems impressive to me for what it handles and all the bullshit that probably gets thrown at it too.

[|iR|Focalor]

It's a good thing, I was beginning to revert back to my pre-internet stone-age instincts. I had thrown all the furniture outside over the back deck rail and was getting ready to set it on fire and dance naked around it with yak shit and deer blood smeared all over my body, a gesture which I seem to remember back in the 80's was used to lure in fertile and receptive females for the mating ritual.

[haunted]

It's a good thing, I was beginning to revert back to my pre-internet stone-age instincts. I had thrown all the furniture outside over the back deck rail and was getting ready to set it on fire and dance naked around it with yak shit and deer blood smeared all over my body, a gesture which I seem to remember back in the 80's was used to lure in fertile and receptive females for the mating ritual.

You do that too when you can't get into dev/random to talk about football?

[|iR|Focalor]

Of corpse! I remember when we all did that before the internet. Before President Bill Clinton invented the internet in 1994, we didn't have yahoo and wikipedia to tell us everything, so we were adamantly convinced that two slot toasters were actually fancy sock drying machines.