packetloss situation (may, 2010)

[console]

Salutations,

ThePlanet is requesting traceroutes from everyone who is experiencing bad packetloss to tastyspleen.net.

I gave them some traces showing packetloss from tastyspleen -> VaeVictis' IP on Friday night.  However, they requested IP addresses of other users who are also experiencing packetloss.  When I provided those IP addresses, they logged onto tastyspleen and ran some traces themselves.  They have pointed out that each of the IPs I provided takes a different route out of theplanet.  Only Vae's went through the NTT.NET provider.

Therefore they are requesting tracroutes from your end, to see if there's any commonality in the reverse route:

We did couple of tests on the network issue and so far we don't see any problems with our NTT provider.

The IPs you provided in you ticket:
24.10.xxx.xxx
62.94.xxx.xxx
216.227.xxx.xxx
204.8.xxx.xxx

Are going out through different providers and we didn't see any loss on those mtr and traceroutes. It is possible that those customers are coming in through the same provide so can you please provide us with mtr or traceroutes from those locations that way we would be able to pint point any network problem. Here are the traceroutes from your server to those locations and you clearly can see that they are going out on different providers:

theplanet@fragbait:~$ traceroute 24.10.xxx.xxx
traceroute to 24.10.xxx.xxx, 30 hops max, 40 byte packets
1 e1.ba.364a.static.theplanet.com (74.54.186.225) 0.467 ms 0.567 ms 0.610 ms
2 te6-6.dsr02.dllstx2.theplanet.com (70.87.254.181) 0.408 ms 0.475 ms 0.542 ms
3 te9-4.dsr02.dllstx3.theplanet.com (70.87.253.125) 0.578 ms 0.657 ms 85.fd.5746.static.theplanet.com (70.87.253.133) 0.751 ms
4 et3-2.ibr03.dllstx3.theplanet.com (70.87.253.17) 0.436 ms 0.554 ms 0.502 ms
5 xe-2-4.r01.dllstx09.us.bb.gin.ntt.net (157.238.225.5) 0.826 ms * *
6 xe-8-2.r02.dllstx09.us.bb.gin.ntt.net (129.250.4.173) 105.662 ms 104.463 ms 104.553 ms
7 po-1.r03.dllstx09.us.bb.gin.ntt.net (129.250.2.94) 0.722 ms 0.808 ms 0.913 ms
8 xe-0.tata-communications.dllstx09.us.bb.gin.ntt.net (129.250.9.14) 6.026 ms 6.124 ms 6.117 ms
9 Vlan554.icore2.DTX-Dallas.as6453.net (206.82.142.22) 1.086 ms Vlan553.icore2.DTX-Dallas.as6453.net (206.82.142.18) 1.136 ms 1.130 ms
10 pos-1-7-0-0-cr01.dallas.tx.ibone.comcast.net (68.86.86.149) 1.466 ms 1.461 ms 1.422 ms
11 pos-1-8-0-0-cr01.denver.co.ibone.comcast.net (68.86.85.177) 17.634 ms 17.618 ms 17.215 ms
12 te-7-4-ar01.nashville.tn.nash.comcast.net (68.86.90.134) 29.685 ms 29.734 ms 29.647 ms
13 te-9-1-ur02.ogden.ut.utah.comcast.net (68.87.170.30) 31.053 ms 31.092 ms 30.798 ms
14 * * *
15 * * *
16 * * *


theplanet@fragbait:~$ traceroute 62.94.xxx.xxx
traceroute to 62.94.xxx.xxx, 30 hops max, 40 byte packets
1 e1.ba.364a.static.theplanet.com (74.54.186.225) 0.440 ms 0.501 ms 0.574 ms
2 te6-6.dsr01.dllstx2.theplanet.com (70.87.254.177) 0.461 ms 0.555 ms 0.585 ms
3 81.fd.5746.static.theplanet.com (70.87.253.129) 0.622 ms te9-4.dsr01.dllstx3.theplanet.com (70.87.253.117) 0.814 ms 0.801 ms
4 25.ff.5746.static.theplanet.com (70.87.255.37) 0.424 ms 0.516 ms 0.544 ms
5 12.87.41.149 (12.87.41.149) 1.622 ms 1.625 ms *
6 cr1.dlstx.ip.att.net (12.122.139.122) 2.199 ms 1.536 ms 1.526 ms
7 ggr6.dlstx.ip.att.net (12.122.139.113) 1.334 ms 1.830 ms 1.825 ms
8 * * *
9 te1-1.ccr02.dfw01.atlas.cogentco.com (66.28.4.129) 178.861 ms 178.967 ms 179.074 ms
10 te8-4.ccr02.iah01.atlas.cogentco.com (154.54.3.178) 71.265 ms 53.743 ms 53.843 ms
11 te3-7.ccr02.atl01.atlas.cogentco.com (154.54.6.137) 104.448 ms 104.460 ms 104.458 ms
12 te0-1-0-2.ccr22.dca01.atlas.cogentco.com (154.54.28.230) 35.879 ms 36.015 ms te0-0-0-3.ccr22.dca01.atlas.cogentco.com (154.54.28.217) 35.763 ms
13 te0-2-0-4.ccr22.fra03.atlas.cogentco.com (154.54.31.242) 127.836 ms 127.848 ms 128.062 ms
14 te1-8.ccr01.str01.atlas.cogentco.com (130.117.49.130) 148.542 ms 148.605 ms 148.337 ms
15 te4-2.ccr01.mil01.atlas.cogentco.com (154.54.1.238) 168.718 ms 168.512 ms 168.801 ms
16 te1-2.ccr01.qzo02.atlas.cogentco.com (130.117.48.86) 146.276 ms 146.318 ms te1-2.ccr01.goa01.atlas.cogentco.com (130.117.48.5) 140.863 ms
17 te1-2.ccr01.rom01.atlas.cogentco.com (130.117.2.10) 284.343 ms 284.480 ms 284.347 ms
18 149.6.22.50 (149.6.22.50) 149.312 ms 148.554 ms 148.484 ms
19 62.94.36.51 (62.94.36.51) 153.311 ms 153.961 ms 152.802 ms
20 ip-145-215.sn-213-198.eutelia.it (213.198.145.215) 163.071 ms 163.277 ms 164.529 ms
21 * * *
22 * * *
23 * * *


theplanet@fragbait:~$ traceroute 216.227.xxx.xxx
traceroute to 216.227.xxx.xxx, 30 hops max, 40 byte packets
1 e1.ba.364a.static.theplanet.com (74.54.186.225) 0.389 ms 0.492 ms 0.549 ms
2 te6-6.dsr02.dllstx2.theplanet.com (70.87.254.181) 0.390 ms 0.469 ms 0.524 ms
3 te9-4.dsr02.dllstx3.theplanet.com (70.87.253.125) 0.561 ms te7-4.dsr02.dllstx3.theplanet.com (70.87.253.121) 0.610 ms 0.701 ms
4 2d.ff.5746.static.theplanet.com (70.87.255.45) 0.424 ms 0.484 ms 0.533 ms
5 12.87.41.149 (12.87.41.149) 1.464 ms 1.461 ms *
6 cr1.dlstx.ip.att.net (12.122.139.122) 50.169 ms 50.136 ms 50.914 ms
7 cr2.nsvtn.ip.att.net (12.122.1.242) 51.136 ms 51.133 ms 51.129 ms
8 cr1.cl2oh.ip.att.net (12.122.28.74) 51.125 ms 51.433 ms 51.431 ms
9 cr2.cl2oh.ip.att.net (12.122.2.126) 50.415 ms 50.422 ms 50.418 ms
10 cr2.phlpa.ip.att.net (12.122.2.210) 50.842 ms 50.840 ms 50.836 ms
11 cr1.phlpa.ip.att.net (12.122.3.225) 50.812 ms 50.149 ms 50.206 ms
12 cr2.cb1ma.ip.att.net (12.122.28.113) 50.530 ms 50.529 ms 50.588 ms
13 gar9.cb1ma.ip.att.net (12.122.98.81) 49.341 ms 49.347 ms *
14 12.90.209.186 (12.90.209.186) 56.563 ms 56.569 ms 56.566 ms
15 dsl-216-227-0-14.pivot.net (216.227.0.14) 57.035 ms 56.748 ms 56.680 ms
16 dsl-216-227-0-142.pivot.net (216.227.0.142) 64.169 ms 63.763 ms 64.659 ms
17 dsl-216-227-0-35.pivot.net (216.227.0.35) 64.460 ms 64.279 ms 64.656 ms
18 * * *
19 * * *
20 * * *


theplanet@fragbait:~$ traceroute 204.8.xxx.xxx
traceroute to 204.8.xxx.xxx, 30 hops max, 40 byte packets
1 e1.ba.364a.static.theplanet.com (74.54.186.225) 0.417 ms 0.532 ms 0.578 ms
2 te6-6.dsr01.dllstx2.theplanet.com (70.87.254.177) 0.510 ms 0.562 ms 0.606 ms
3 te7-4.dsr01.dllstx3.theplanet.com (70.87.253.113) 0.694 ms 81.fd.5746.static.theplanet.com (70.87.253.129) 0.771 ms 0.852 ms
4 25.ff.5746.static.theplanet.com (70.87.255.37) 0.396 ms 0.468 ms 0.520 ms
5 GigabitEthernet7-3.ar2.DAL2.gblx.net (64.208.170.197) 0.612 ms TenGigabitEthernet6-2.ar5.DAL2.gblx.net (67.17.168.93) 44.921 ms GigabitEthernet7-3.ar2.DAL2.gblx.net (64.208.170.197) 0.644 ms
6 208.178.62.50 (208.178.62.50) 56.814 ms 56.790 ms 56.645 ms
7 208-84-198-14.rev.ackley.net (208.84.198.14) 56.943 ms 57.844 ms 56.985 ms
8 v72-cr1.bluemtn.ackley.net (208.84.198.10) 58.054 ms 58.056 ms 58.244 ms
9 v49-br1.prn.ackley.net (208.84.198.18) 58.687 ms 58.670 ms 58.671 ms
10 204.11.158.202 (204.11.158.202) 60.772 ms 59.836 ms 61.634 ms
11 * at2-0-0-1201-155M.br2.crwn.ackley.net (204.11.155.162) 66.768 ms 66.771 ms
12 * * *
13 * * *
14 * * *


Couple of things that you should look into when you are experiencing the packet loss:

Ask a customer to run mtr or traceroute to your server.
Here is a link for WINMTR http://www.snapfiles.com/get/winmtr.html.

Note, I downloaded the WINMTR tool mentioned above, however, it doesn't work for me.  I even downloaded the source code and recompiled it and have debugged it a bit, but, I'm not sure yet why it's not working. (It's actually sending ICMP echo requests out the port but getting no response... I can see using tcpdump they're essentially similar to the ICMP echo requests sent by the Ping tool, but, maybe WINMTR's are malformed in some way.)


Anyway...

As a first step, if anyone experiencing bad packetloss to tastyspleen.net could please run some traceroutes, and post the results here.


Thanks,

[VaeVictis]

I can post mine if you need it, dunno if you do... Here is just some people that have the problem:

Theres me of course...

Onphone also lives in utah, he has the same packet loss as me i believe

tailgunner said he was having packet loss issues to a different server (irtdm i think) but it was the same subnet so i think its also in theplanet's network

Erica has been having more packet loss issues than usual lately if i remember correctly, ingame name is Er!c@

pepp5 seemed to be having troubles also, dunno if its related, hes located in WA i think

[l0ckp1ck]

i have attached a txt file of mine. over the past week i seem to be getting alot of packet loss and my ping sky rockets at a certain time of the day. i normally ping 210-220 but at around the same time every day over the past week it my ping jumps to nearly 400 at times. i know connecting from australia i am bound to have some issues but it seems to have gotten worse over the past week or so and i have spoken to my isp and they assure me all is fine on my end

[Slayer :D]

That sounds almost exactly like what I have, lockpick. One moment I'm pinging 50 (normal) and the next it leaps up to 200, and then drops back down again!

[[BTF]Sigma]

Checked it out real quick on MUTANT and VANILLA and I'm not getting any packet loss......

Here's the tracert:

Tracing route to tastyspleen.net [74.54.186.226]
over a maximum of 30 hops:

  1     *        *        *     Request timed out.
  2    17 ms     8 ms     7 ms  rd2bb-ge0-0-0-3.vc.shawcable.net [64.59.152.66]
  3     8 ms     7 ms     7 ms  rc2bb-tge0-2-2-0.vc.shawcable.net [66.163.69.133]
  4    31 ms    29 ms    35 ms  rc3sj-tge0-0-1-0.cl.shawcable.net [66.163.77.146]
  5    40 ms    39 ms    39 ms  eq-sanjose-10G.theplanet.com [206.223.116.148]
  6    71 ms    71 ms    69 ms  c2.fd.5746.static.theplanet.com [70.87.253.194]
  7    70 ms    71 ms    72 ms  te9-2.dsr02.dllstx3.theplanet.com [70.87.253.30]
  8     *        *        *     Request timed out.
  9    70 ms    71 ms    71 ms  te1-2.car06.dllstx6.theplanet.com [70.87.254.182]
 10    79 ms    79 ms    79 ms  tastyspleen.net [74.54.186.226]
Trace complete.


Always have been getting 80ms to TS. Seems stable.

[Punk_FAS]

I noticed my ping just jumped up 25-30 ms.  Not sure about packet loss, but I guess the route has changed =(

[reaper]

The packet can take a different route to the destination, and back from the destination. 

Where exactly is the packet loss?  Is it inside ThePlanet's network, or in NTT, Comcast??  To identify this, without working with the providers, you will have to look at something like pathping (traceroute), or somehow get a reply from just past the routers.  There are some technicalities to looking at the latency values of ICMP time to live exceeded messages, or UDP port unreachables in the responses (your packet could be a lower priority quality of service wise, or the control-plane of the router (packet not switced/routed in specialized hardware) could be forwarding the packet), and you are also sending one packet, yet you are checking for four percent loss.  You could use pathping, or even a script to run traceroute a bunch of times, to get nice information.

The traceroute gets the information by messing with the time-to-live value, but you can use UDP,TCP, or ICMP to get responses.  The WinMRT tool is really nothing special, there's not a whole lot more you can do other than what pathping and WinMRT are doing that doesn't require manual intervention to test connectivity and loss through each hop without the provider.  That said, you should be able to get all the traceroutes and pathpings, and say, this packet loss is very lkely in NTT, or going from Comcast to NTT.  The Planet would then work with them to fix the issue, most likely routing needs to be changed or congestion resolved, because there's just to many packets in the queues.  The Planet can alter the route of the tastyspleen servers as seen from the internet, so the traffic doesn't go to the problem provider.  The issue with that is ISPs will sometimes not accept routers smaller than /24 in the routing tables, so traffic is that whole network, will not directly connect to NTT, although I'm not sure how big of a deal that is.  If there are common providers at the source to, you could possibley do something like get NTTs routes out the problem circuits and point them somewhere else at ThePlanet, but it'd be good to see what's going on with the problem provider.  They can probably also route you, based on any of your source networks and size doesn't matter, only out one provider, or others, and just skip NTT out of the mix for you completely.

You could also test with iperf, which is avaiable on windows. You can control everything, so you could say, use this TCP window, transfer this file, and note all the retransmissions from the dropped packets.  Most likely they don't have enough bandwidth for gaming, or something is messed up.

[Slayer :D]

LMAO lockpick you're coming through level3.net too!!!

reaper, I'm a newbie with respect to this--where is my ping jumps occuring?

[reaper]

Run traceroute a bunch or pathping, or the tool posted and see where the packets drop (*'s) are, and what hops have the high ping times.  That's the first thing you have to do is see where the packet loss is occuring, the packet loss is bad, and you should be able to find where it is.  Hopefully we can see loss, and find out where it is.  Remember there is 4 percent loss and traceroute is sending one packet to each hop.

you're also on dialup, who knows if your line dropped the packet on the way back.  basically we are trying to find the consistent 4 to 8 percent loss, so we'll need some pathpings or traceroutes, and as good as information as possible.  running traceroute a bunch with a batch file would be nice, or a the tool posted or pathping

The internet is not really designed for such low latency anywhere and everywhere.  ThePlanet is near ISP points of presence (and has their own nice network, that they can fix if we find the loss there), and the large ISPs peer with so many networks, and improve the peerings.  You basically pick the route with the least amount of ISPs you are going through (if you want to see a complicated algorithm, search for BGP path selection Cisco, but unless changed, it will almost always choose the least amount of autonomous systems to pick the best route, and one ISP is usually one AS), or a financially related route, and that is the speed you get.  It's not perfect, but considering the communication is possible through packet switching, and not circuit switching, that's what you get, and you get any to any communication.

edit again : ),
btw, internal to the ISPs , your packet may use an interal routing protocol, and latency may be a metric.  When in Level3, the routing protocol is BGP (iBGP and eBGP), and BGP doesn't factor in latency, but it may at ThePlanet.  But it's not like you can just nonchalantly add latency as the primary metric in BGP, it would have devastating effect.  But there are devices ThePlanet may have that choose the best upstream based on latency, maybe the gaming ISPs use these devices.  I"m not to familar with it, generally everything is really high speed, because the internet is fast : ).

[Barton]

No problems here.

Tracing route to tastyspleen.net [74.54.186.226]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  192.168.0.1
  2     9 ms    12 ms     9 ms  10.151.64.1
  3    10 ms     8 ms    10 ms  csw02rochmn-gbe-4-18.roch.mn.charter.com [96.34.25.132]
  4     9 ms     8 ms     9 ms  96-34-25-164.static.unas.mn.charter.com [96.34.25.164]
  5    11 ms    12 ms    16 ms  96-34-17-2.static.unas.mn.charter.com [96.34.17.2]
  6    19 ms    13 ms    13 ms  crr01euclwi-gbe-6-0-0.eucl.wi.charter.com [96.34.16.145]
  7    13 ms    13 ms    13 ms  96-34-16-228.static.unas.wi.charter.com [96.34.16.228]
  8    13 ms    13 ms    27 ms  bbr02euclwi-tge-0-0-0-6.eucl.wi.charter.com [96.34.2.78]
  9    24 ms    25 ms    24 ms  96-34-0-100.static.unas.mo.charter.com [96.34.0.100]
 10    33 ms    25 ms    25 ms  prr01chcgil-tge-3-2.chcg.il.charter.com [96.34.3.57]
 11    25 ms    25 ms    24 ms  be-10-805-pe01.350ecermak.il.ibone.comcast.net [75.149.231.205]
 12    32 ms    25 ms    27 ms  pos-1-6-0-0-cr01.chicago.il.ibone.comcast.net [68.86.86.77]
 13    51 ms    51 ms    52 ms  pos-2-15-0-0-cr01.atlanta.ga.ibone.comcast.net [68.86.85.166]
 14    71 ms    70 ms    71 ms  pos-1-10-0-0-cr01.dallas.tx.ibone.comcast.net [68.86.86.129]
 15    74 ms    70 ms    73 ms  pos-0-1-0-0-pe01.1950stemmons.tx.ibone.comcast.net [68.86.86.94]
 16    50 ms    48 ms    49 ms  theplanet-cr01.dallas.tx.ibone.comcast.net [75.149.228.2]
 17    54 ms    52 ms    48 ms  te9-1.dsr01.dllstx3.theplanet.com [70.87.253.6]
 18    50 ms    59 ms    48 ms  te1-4.dsr01.dllstx2.theplanet.com [70.87.253.114]
 19    49 ms    48 ms    48 ms  te1-1.car06.dllstx6.theplanet.com [70.87.254.178]
 20    49 ms    49 ms    49 ms  tastyspleen.net [74.54.186.226]

Trace complete.

[reaper]

No loss here, maybe try the problem ISP and location?  Maybe just more data from the people that reported the issues will do it.


  1  ---     100/ 100 =100%   100/ 100 =100%  66.174.121.1
                                0/ 100 =  0%   |
  2  ---     100/ 100 =100%   100/ 100 =100%  63.sub-66-174-121.myvzw.com [66.17
4.121.63]
                                0/ 100 =  0%   |
  3  ---     100/ 100 =100%   100/ 100 =100%  193.sub-69-83-15.myvzw.com [69.83.
15.193]
                                0/ 100 =  0%   |
  4  ---     100/ 100 =100%   100/ 100 =100%  18.sub-69-83-14.myvzw.com [69.83.1
4.18]
                                0/ 100 =  0%   |
  5  ---     100/ 100 =100%   100/ 100 =100%  132.sub-66-174-18.myvzw.com [66.17
4.18.132]
                                0/ 100 =  0%   |
  6  ---     100/ 100 =100%   100/ 100 =100%  131.sub-66-174-18.myvzw.com [66.17
4.18.131]
                                0/ 100 =  0%   |
  7  ---     100/ 100 =100%   100/ 100 =100%  253.sub-69-83-3.myvzw.com [69.83.3
.253]
                                0/ 100 =  0%   |
  8  183ms     0/ 100 =  0%     0/ 100 =  0%  te2-3-0d0.cir1.nyc-ny.us.xo.net [2
09.220.16.89]
                                0/ 100 =  0%   |
  9  207ms     0/ 100 =  0%     0/ 100 =  0%  206.111.13.54.ptr.us.xo.net [206.1
11.13.54]
                                0/ 100 =  0%   |
 10  238ms     0/ 100 =  0%     0/ 100 =  0%  The-Planet.TenGigabitEthernet2-3.a
r2.HOU1.gblx.net [64.214.196.58]
                                0/ 100 =  0%   |
 11  ---     100/ 100 =100%   100/ 100 =100%  et5-4.ibr03.dllstx3.theplanet.com
[70.87.253.49]
                                0/ 100 =  0%   |
 12  ---     100/ 100 =100%   100/ 100 =100%  te7-1.dsr02.dllstx3.theplanet.com
[70.87.253.18]
                                0/ 100 =  0%   |
 13  ---     100/ 100 =100%   100/ 100 =100%  86.fd.5746.static.theplanet.com [7
0.87.253.134]
                                0/ 100 =  0%   |
 14  ---     100/ 100 =100%   100/ 100 =100%  te1-2.car06.dllstx6.theplanet.com
[70.87.254.182]
                                0/ 100 =  0%   |
 15  252ms     0/ 100 =  0%     0/ 100 =  0%  tastyspleen.net [74.54.186.226]

[Slayer :D]

I ran a number of tracert's and this is a typical response. Also, I posted a "ping" request taken at about the same time . I seem to always have packet loss in my ISP (core.fuse.net). There is nothing wrong in the ping times for ThePlanet.com for me.

[reaper]

there may be loss just a few hops in, what happens if you run something like:
ping google.com -t
ping tastyspleen.net -t
and you run that for a few minutes, what does it say at the end?  looks like you are on dialup, so the line might not be the best

[Slayer :D]

I ran tracert several times from around 15:00 to 21:00 last night. I noticed packet loss is always in the *.core.fuse.net router.
I am not on dialup (though when I went to fuse.net's homepage it does seem to be a "5x speed" dialup provider ). I am on a 256 kbps down/64 kbps up DSL connection.

I tried running pathping, but every time it gets to *.core.fuse.net it drops out the connection.

[Slayer :D]

LOL look what I just found while looking around on the web to find out if anyone else had problems with core.fuse.net!

I guess this explains why some days my connection is really good and other days it is terrible!