east.tastyspleen.net outage (hurricane Sandy)

[quadz]

Received the following info this evening from the hosting provider:

**EMERGENCY NOTIFICATION**
 
Please be advised that Internap's LGA11 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged and destroyed the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost. We urge all Internap customers to take necessary remote action, if shutting down equipment in a more graceful fashion is possible before the outage occurs.
 
The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.

So: east.tastyspleen.net has been backed up and shut down cleanly for the duration.

GOTO JUMP has been relocated to Dallas for the time being (all map times should be intact.)


^{“ ”}

[VaeVictis]

see, this shit doesn't happen with datacenters in utah... lol

[Sgt. Dick]

If a basement floods in Utah, you have to get a new mattress 

[VaeVictis]

If a basement floods in Utah, you have to get a new mattress 

nah, statistical fact that utah is a sweet spot for things like datacenters due to the extremely low natural disaster rate compared to the rest of the country

[adz1La]

too bad everyone in utah is a fucking faggot (bundle of sticks)

[fdrjk]

If a basement floods in Utah, you have to get a new mattress 

nah, statistical fact that utah is a sweet spot for things like datacenters due to the extremely low natural disaster rate compared to the rest of the country

Yes, because New Jersey is always getting slammed by hurricanes.

[|iR|Focalor]

too bad everyone in utah is a fucking faggot (bundle of sticks)

And mormon, which is even worse.

[haunted]

that is not what utah is known for concerning data centers....

utah is known to be next to colorado, which is actually known for quality/quanity and strategically placed away from natural disasters. Utah isn't known for it. New york is more popular than colorado anyway.

[|iR|Focalor]

I don't think it's really any concern for natural disasters when they build datacenters. It's basically all about location within the global network of telecommunications. New York and Atlanta are going to be the two most likely places on the east coast to place any type of datacenter because those are the two main points at which existing communication lines go overseas. Between Dallas and the west coast, Denver is probably the next closest major communications hub before reaching Los Angeles.

[haunted]

I remember reading how climate facilitated cooling these facilities a lot which is why toronto and new york are obvious choices.. so "the mile high" city would be an obvious spot climate-wise, except with less risk of natural disasters. but it's not a big thing of emphasis like foc said.

[|iR|Focalor]

Regional climate isn't a problem, these places have the best AC systems you'll ever find. I did some HVAC work on a couple of commercial jobs that were datacenters. One was a huge brand new AT&T datacenter in Alpharetta GA. When I first arrived to help out the the crew that had already been there a couple of months, it blew my fucking mind to see how huge the system was. The building itself was enormous, basically a huge warehouse. Their was another contractor on site installing a raised flooring system which allowed cold ventilation to enter up through the floors as well as allow for wiring to be run all over the place once the server racks and servers had been placed. The AC and sprinkler system was not your typical setup. It was all 100% glycol, which is to ensure that if there is some type of incident like a fire or a busted pipe, water will not create any type of electrocution hazards or damage any of the equipment and electronics. I ended up having to go back there to do some finishing touches on rooftop units, and I can tell you that by the time they got all the virtual miles of rows of computer racks in there and had the AC system running, the inside of that place felt like the god damned North Pole. It was hot as shit outside that day, and I was going to be spending a few hours on the roof... but the security manager completely forbid me from taking ANYTHING containing water through the server area where the stairs to the roof hatch was located. I ended up having to throw a rope over the side of the building and pull up my ice chest that way.

[haunted]

well yeah, but I assumed all of that. I didn't indicate this precisely, but I was thinking about it in terms of cost efficiency. The annual cost to ventilate one of these buildings in toronto with ac has GOT to be way cheaper than in Atlanta. Seemed like a safe assumption, regarding that they are commercial/industrial-sized AC units working on overdrive as you said.

[fdrjk]

The routing for east.tastyspleen.net seems to be on the fritz.

  1    <1 ms    <1 ms    <1 ms 
  2    35 ms    12 ms    26 ms  xxx.hsd1.md.comcast.net [69.251.11.1

  3    10 ms     9 ms    10 ms  te-8-1-ur01.westminster.md.bad.comcast.net [68.8
7.136.113]
  4    16 ms    11 ms    28 ms  te-1-3-ur02.westminster.md.bad.comcast.net [69.1
39.175.18]
  5    10 ms    11 ms    11 ms  xe-8-0-1-0-ar04.whitemarsh.md.bad.comcast.net [6
9.139.175.21]
  6    19 ms    19 ms    17 ms  pos-3-4-0-0-cr01.newyork.ny.ibone.comcast.net [6
8.86.94.61]
  7    30 ms    16 ms    16 ms  pos-1-0-0-0-pe01.111eighthave.ny.ibone.comcast.n
et [68.86.86.42]
  8    16 ms    16 ms    16 ms  er2-tengig3-3.newyork.savvis.net [208.173.138.13
]
  9    30 ms    17 ms    17 ms  208.173.129.62
 10    18 ms    16 ms    16 ms  border3.po1-20g-bbnet1.nym008.pnap.net [216.52.9
5.4]
 11    23 ms    16 ms    16 ms  inapvoxcust-8.border3.nym008.pnap.net [63.251.26
.2]
12   189 ms    73 ms   237 ms  0.ae59.tsr1.lga3.us.voxel.net [208.122.44.201]
 13    37 ms    38 ms    38 ms  4000.te1-49.esr2.ldj1.us.voxel.net [173.231.161.
2]
 14    39 ms    37 ms    38 ms  vicious.voxel.net [69.9.166.38]
 15    39 ms    37 ms    38 ms  69.9.168.4

Trace complete.

[VaeVictis]

just fyi

[haunted]

just fyi, you said "statistical fact that utah is a sweet spot for things like datacenters". That's wrong, fyi.

lol lol