Author Topic: Unplanned Server Outage - 21aug2014  (Read 3145 times)

Offline quadz

  • Brobdingnagian Member
  • ***
  • Posts: 4252
    • View Profile
  • Rated:
Unplanned Server Outage - 21aug2014
« on: August 21, 2014, 10:59:38 PM »
Salutations,

Here's a synopsis of the unplanned server outage that occurred today.

At 1330 days of uptime, our primary server fragbait.tastyspleen.net crashed with a kernel panic in the Ethernet driver.

Subsequent attempts to reboot and/or power cycle the server were met with failures to bring up the network driver:

[    1.921773] e1000e: Intel(R) PRO/1000 Network Driver - 1.0.2-k2
[    1.921836] e1000e: Copyright (c) 1999-2008 Intel Corporation.
[    1.921945] e1000e 0000:04:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[    1.922025] e1000e 0000:04:00.0: setting latency timer to 64
[    1.922140] e1000e 0000:04:00.0: irq 54 for MSI/MSI-X
[    2.172397] e1000e 0000:04:00.0: PCI INT A disabled
[    2.172467] e1000e: probe of 0000:04:00.0 failed with error -2
[    2.172552] e1000e 0000:04:00.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19
[    2.172629] e1000e 0000:04:00.1: setting latency timer to 64
[    2.172754] e1000e 0000:04:00.1: irq 54 for MSI/MSI-X
[    2.423008] e1000e 0000:04:00.1: PCI INT B disabled
[    2.423070] e1000e: probe of 0000:04:00.1 failed with error -2


After I opened a support ticket, I was able to watch the tech assigned by SoftLayer over the virtual KVM as he attempted to troubleshoot.

Sometimes when the tech would boot into a "rescue" kernel, the network driver would load. Sometimes after that when he booted back into the fragbait kernel the networking would still be OK--which is why we had a brief interval in the afternoon where the system was back up for awhile. However, on subsequent reboots the networking would go back to the error condition above.

The tech suggested updating the BIOS on the motherboard, but unfortunately this had no noticable effect on the problem.

The tech then continued running various diagnostic programs at occasional intervals, however the programs the tech was running depended on the Ethernet devices existing (but the devices did not exist, because the driver was unable to load, again related to the error above -- so these diagnostic tools were unable to run.)

So it was an interesting experience watching sometimes lengthy periods of inactivity punctuated by the running of another diagnostic tool which had no chance of working.

Eventually the problem was escalated to a higher tier tech who was aware of a rather bizarre problem with the particular motherboard on the fragbait server.

He was able to get the networking back up, and explained the issue as follows:

Quote
I am going to go ahead and put an account note on your account so that we can get this fixed in a faster fashion in the future.

There is a specific issue with the motherboard in this server. There is a trick of sorts that has to be performed as this motherboard can lose network access at times.

basically for the server fragbait2 which has the X7DBR-E we have to shutdown the machine, set the network interfaces to 0 then push the config to our switches.

Then wait a minute or two, then power back on the machine and then reset the speed of the interfaces back to the required speed which for this one is 1Gbps.

After that networking starts to work and is a documented issue for this board.

I did go ahead and add the notes for this server so hopefully any issues will be fixed in a quick manner.

So.... that's definitely an odd one. I'm glad somebody there knew a workaround. It will be interesting to see how long it lasts. (If it can last another 1330 days, I won't complain much. But who knows...)


:exqueezeme:
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus
"He knew all the tricks, dramatic irony, metaphor, bathos, puns, parody, litotes and... satire. He was vicious."

Offline VaeVictis

  • i was -1 because you fucking suck
  • Brobdingnagian Member
  • *
  • Posts: 4467
    • View Profile
  • Rated:
Re: Unplanned Server Outage - 21aug2014
« Reply #1 on: August 22, 2014, 01:22:36 AM »
Never seen such an odd issue with a super micro board like that. Usually super micro boards are rock solid without any kind of issues
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

Offline fishxz

  • Newbie
  • *
  • Posts: 28
    • View Profile
  • Rated:
Re: Unplanned Server Outage - 21aug2014
« Reply #2 on: August 22, 2014, 05:53:58 AM »
sad to hear quadz :( my heart is bleeding for ur uptime :(
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

Offline bluemeanies

  • Swanky Member
  • *****
  • Posts: 519
    • View Profile
  • Rated:
Re: Unplanned Server Outage - 21aug2014
« Reply #3 on: August 22, 2014, 10:50:02 AM »
1330

I think you misspelled 1337. Thanks for all your hard work and free service quadz.

And yes nerds...we've all heard the story of that company finding a server that had been up for 20 years dry walled in behind a wall in some basement. That uptime seems impressive to me for what it handles and all the bullshit that probably gets thrown at it too.
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus
Quote from: John Kreese
Mercy is for the weak. Here, in the streets, in competition: A man confronts you, he is the enemy. An enemy deserves no mercy.

Online |iR|Focalor

  • Irrepressibly Profuse Member
  • *
  • Posts: 13689
  • OFFO
    • View Profile
    • Focalor's Lair @ Tastyspleen.net
  • Rated:
Re: Unplanned Server Outage - 21aug2014
« Reply #4 on: August 22, 2014, 12:48:31 PM »
It's a good thing, I was beginning to revert back to my pre-internet stone-age instincts. I had thrown all the furniture outside over the back deck rail and was getting ready to set it on fire and dance naked around it with yak shit and deer blood smeared all over my body, a gesture which I seem to remember back in the 80's was used to lure in fertile and receptive females for the mating ritual.
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

Offline haunted

  • Inexorably Palaverous Member
  • ******
  • Posts: 8425
  • I am hollywood.
    • View Profile
  • Rated:
Re: Unplanned Server Outage - 21aug2014
« Reply #5 on: August 22, 2014, 01:09:16 PM »
It's a good thing, I was beginning to revert back to my pre-internet stone-age instincts. I had thrown all the furniture outside over the back deck rail and was getting ready to set it on fire and dance naked around it with yak shit and deer blood smeared all over my body, a gesture which I seem to remember back in the 80's was used to lure in fertile and receptive females for the mating ritual.

You do that too when you can't get into dev/random to talk about football?
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

Online |iR|Focalor

  • Irrepressibly Profuse Member
  • *
  • Posts: 13689
  • OFFO
    • View Profile
    • Focalor's Lair @ Tastyspleen.net
  • Rated:
Re: Unplanned Server Outage - 21aug2014
« Reply #6 on: August 22, 2014, 01:41:26 PM »
Of corpse! I remember when we all did that before the internet. Before President Bill Clinton invented the internet in 1994, we didn't have yahoo and wikipedia to tell us everything, so we were adamantly convinced that two slot toasters were actually fancy sock drying machines.
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

 

El Box de Shoutamente

Last 10 Shouts:

 

quadz

May 27, 2020, 01:20:31 PM
(launch scrubbed due to weather)

-Unh0ly-

May 22, 2020, 01:44:29 AM
NEW WEBSITE
http://www.unholyq2.com
NEW WEBSITE
 

Admin

May 21, 2020, 02:46:32 PM
(an OS upgrade last weekend broke the shoutbox - working again now)

Punk_FAS

May 14, 2020, 12:10:28 PM
 

Prophet

May 03, 2020, 11:35:42 AM
Moo moo? MooOooooo
 

Waffle Whiffer

May 03, 2020, 08:19:10 AM
Moo
 

Reefer

April 27, 2020, 05:15:19 PM
2dum thank you — I will be keeping your awesome RA2 map guide handy! Not to infringe your copyright, but I am 2dum to find the 2nd church, even with what is probably a screenshot of it. TY again will keep combing!
 

2d

April 25, 2020, 03:46:23 PM
Reefer: http://forum.tastyspleen.net/quake/index.php?topic=21180.0

7th post on page 1, i made an ra2 map list a while ago, should have all maps ;)
 

|iR|Focalor

April 20, 2020, 02:14:45 AM


Why yes, it is indeed the greatest movie ever.

Show 50 latest
Welcome, Guest. Please login or register.
May 29, 2020, 10:18:50 AM

Login with username, password and session length