Server Lag and Lockups

[otg]

HI, i keep getting random locking and persistent lag. i did a trace it appears a timeout happens at hop 10.

1    <1 ms    <1 ms    <1 ms  otg [192.168.1.1]
  2    12 ms    11 ms    10 ms  66.185.91.106
  3     9 ms    12 ms    13 ms  xxxxxxx.cpe.teksavvy.com [xxxxx]
  4    11 ms    10 ms    10 ms  xxxxx.cpe.teksavvy.com [xxxxxx]
  5    29 ms    12 ms    10 ms  xxxxxxx-tor2.teksavvy.com [xxxxxxxx]
  6    11 ms    10 ms     9 ms  xxxxx-tor.teksavvy.com [xxxxxxx]
  7    11 ms     9 ms    10 ms  xxxxxx.cr1-tor1.ip4.gtt.net [xxx]
  8    47 ms    49 ms    49 ms  et-0-0-71.cr11-dal3.ip4.gtt.net [89.149.139.201]
  9    75 ms    47 ms    49 ms  ip4.gtt.net [173.205.47.98]
 10     *        *        *     Request timed out.
 11    49 ms    49 ms    48 ms  Dallas-TX.r19.Public.Pwr-2xPDU-2xUPS-2N-100SLA.incero.com [144.168.34.82]
 12    48 ms    49 ms    52 ms  tastyspleen.net [23.227.170.222]

[Admin]

HI, i keep getting random locking and persistent lag. i did a trace it appears a timeout happens at hop 10.

Hi, thanks for posting a traceroute.

A tool like WinMTR that (in effect) combines ping and traceroute may be more helpful:

http://winmtr.net/download-winmtr/

This type of tool will gather statistics for awhile, and can illuminate intermittent problems that would escape a standard traceroute (which only pings each host 3 times.)

Note that the hop 10 timeout shown in your traceroute does not in itself indicate a problem, because hops 11 and 12 are unaffected. (Some routers won't 'participate' in pings and traceroutes (refusing to return TTL-expired packets to the sender) and so we have to look at subsequent hops to see whether it's affecting anything further down the line. If nothing further along is affected, then it's a false positive.)

Anyway, WinMTR will ping the whole route until you tell it to stop. I'd recommend collecting at least 1000 pings.

I've also run an mtr from the server to your IP. I am seeing strange behavior in hops 8 and 9. But I can't be sure it's a real problem because your local router isn't answering pings:

                               My traceroute  [v0.87]
ts-main (0.0.0.0)                                          Sun Dec  2 16:25:38 2018
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                           Packets               Pings
 Host                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 23.227.170.209                        0.0%  1008    0.7   0.8   0.5  10.0   0.6
 2. ???
 3. 144.168.32.1                          0.0%  1008    0.5   1.1   0.4  51.4   4.2
 4. ae32.cr11-dal3.ip4.gtt.net            0.0%  1008    0.5   1.7   0.4  39.2   3.8
 5. et-8-3-0.cr1-tor1.ip4.gtt.net         0.0%  1008   36.5  36.5  36.4  54.4   1.2
 6. ip4.gtt.net                           0.0%  1008   36.6  37.3  36.5  91.6   4.0
 7. ae4-0-agg01-tor.teksavvy.com          0.0%  1008   36.8  37.0  36.4  57.6   2.2
    ae1-0-agg01-tor2.teksavvy.com
    ae8-0-bdr01-tor2.teksavvy.com
 8. 104-195-129-22.cpe.teksavvy.com       5.6%  1008   37.5 891.3  36.5 12485 3132.
    ae1-0-agg01-tor.teksavvy.com
    ae3-0-agg01-tor2.teksavvy.com
    104-195-128-134.cpe.teksavvy.com
 9. 104-195-129-22.cpe.teksavvy.com      56.7%  1008  12403 2950.  37.0 12527 5245.
    104-195-128-134.cpe.teksavvy.com
10. ???


The above was a trace using UDP packets (like Q2 uses.)

In a second run, the results are slightly more "sane" using ICMP packets, though still showing some 'apparent' loss in hop 8.


                               My traceroute  [v0.87]
ts-main (0.0.0.0)                                          Sun Dec  2 16:30:55 2018
Resolver: Received error response 2. (server failure)er of fields   quit
                                           Packets               Pings
 Host                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 23.227.170.209                        0.0%  1011    0.8   0.8   0.5   7.6   0.6
 2. ???
 3. 144.168.32.1                          0.0%  1011    0.5   1.1   0.4  40.4   4.3
 4. ae32.cr11-dal3.ip4.gtt.net            0.0%  1011    1.3   2.3   0.4  55.5   5.0
 5. et-8-3-0.cr1-tor1.ip4.gtt.net         0.0%  1011   37.4  36.5  36.4  54.6   1.2
 6. ip4.gtt.net                           0.0%  1011   36.5  36.8  36.4  70.2   2.5
 7. ae4-0-agg01-tor.teksavvy.com          0.0%  1011   36.5  36.5  36.4  57.9   1.9
 8. 104-195-128-134.cpe.teksavvy.com      4.9%  1011   37.4  37.0  36.9  38.7   0.1
 9. ???


In any case, it doesn't prove anything without the final hop. If you can edit the settings in your local router (or cable modem) to enable responding to pings, we can get better stats from the server side.

We can compare these to the WinMTR stats from your side.


Regards,

quadz

[otg]

I'm having issues logging into this forum, each time i try it says "session expired." Anyways... was playing this morning and there was a delay in my shooting as usual... here are the stats.


|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                                   omega -    0 | 1476 | 1476 |    0 |    0 |   19 |    0 |
|                             66.xx.xx.xx -    0 | 1476 | 1476 |    8 |   14 |   95 |   53 |
|        104-195-128-134.cpe.teksavvy.com -   16 |  914 |  773 |    7 |    8 |   54 |    9 |
|         104-195-129-21.cpe.teksavvy.com -    0 | 1476 | 1476 |    7 |   15 |   88 |   10 |
|          ae10-0-bdr01-tor2.teksavvy.com -    0 | 1476 | 1476 |    7 |   10 |   34 |    8 |
|           ae12-0-bdr01-tor.teksavvy.com -    0 | 1476 | 1476 |    7 |   10 |   74 |    7 |
|           ae16-104.cr1-tor1.ip4.gtt.net -    0 | 1476 | 1476 |    7 |   10 |   50 |    9 |
|         et-0-0-71.cr11-dal3.ip4.gtt.net -    0 | 1476 | 1476 |   43 |   46 |   85 |   45 |
|                             ip4.gtt.net -    0 | 1476 | 1476 |   43 |   46 |   99 |   57 |
|                   No response from host -  100 |  296 |    0 |    0 |    0 |    0 |    0 |
|Dallas-TX.r19.Public.Pwr-2xPDU-2xUPS-2N-100SLA.incero.com -
|                                              0 | 1476 | 1476 |   44 |   45 |   71 |   44 |
|                         tastyspleen.net -    0 | 1476 | 1476 |   43 |   45 |   66 |   44 |
|________________________________________________|______|______|______|______|______|______|

   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider



(Edit by Admin: formatting)

[Admin]

Unfortunately (for our diagnostic purposes) the connection quality looks great in that trace. (The packetloss seen at the 104-195-128-134.cpe.teksavvy.com hop is a red herring, because it's not affecting subsequent hops.)

Only other thing I can think of is to leave the WinMTR trace running WHILE you play Q2, and hope you encounter the in-game lag during that time. Hopefully then we might also see it in the trace.



Regarding forum login:

Try clearing browser cookies for *.tastyspleen.net, then use this login interface:

http://forum.tastyspleen.net/quake/index.php?action=login

And check "Always stay logged in"

[otg]

Okay i will try that thanks.

[|iR|Focalor]

I dunno about locking, but the lagging... yeah, I've noticed it too. Usually my netgraphs got at least 3 to 6 red spikes on the screen at one time, spread out and never constant enough to make red bars, but consistent enough to be VERY annoying, and usually gets worse as population on a server increases. If they claim you're paying for "better" service, I've yet to see evidence of it.

I've also noticed that the admin and quadz avatars don't show up on the forums anymore, haunteds too I believe. Maybe those were loaded into the forums memory and lost during the transitition?

[otg]

Here is todays update while i was playing. I always have this lag everytime i shoot there is a 2 second delay, it is very annoying.


|        104-195-128-134.cpe.teksavvy.com -   21 |  815 |  650 |    5 |    8 |   56 |    7 |

|         104-195-129-21.cpe.teksavvy.com -    0 | 1474 | 1474 |    5 |    9 |   41 |    8 |

|          ae10-0-bdr01-tor2.teksavvy.com -    0 | 1474 | 1474 |    5 |   10 |   42 |    9 |

|           ae12-0-bdr01-tor.teksavvy.com -    0 | 1474 | 1474 |    4 |   10 |   54 |    8 |

|           ae16-104.cr1-tor1.ip4.gtt.net -    0 | 1474 | 1474 |    5 |    9 |   47 |   12 |

|         et-0-0-71.cr11-dal3.ip4.gtt.net -    0 | 1474 | 1474 |   43 |   48 |  433 |   46 |

|                             ip4.gtt.net -    0 | 1474 | 1474 |   41 |   48 |   93 |   44 |

|                   No response from host -  100 |  294 |    0 |    0 |    0 |    0 |    0 |

|Dallas-TX.r19.Public.Pwr-2xPDU-2xUPS-2N-100SLA.incero.com -    0 | 1474 | 1474 |   42 |   47 |   69 |   44 |

|                         tastyspleen.net -    0 | 1474 | 1474 |   43 |   46 |   75 |   47 |

|________________________________________________|______|______|______|______|______|______|

[Admin]

Here is todays update while i was playing. I always have this lag everytime i shoot there is a 2 second delay, it is very annoying.

That is really strange. Thanks for the trace. I don't know what to make of it. The trace looks fine.

 

When the lag happens does it show up on the in-game netgraph (or whatever it's called in q2pro these days?)

[Admin]

I dunno about locking, but the lagging... yeah, I've noticed it too. Usually my netgraphs got at least 3 to 6 red spikes on the screen at one time, spread out and never constant enough to make red bars, but consistent enough to be VERY annoying, and usually gets worse as population on a server increases. If they claim you're paying for "better" service, I've yet to see evidence of it.

Hmm. 

For better or worse, one thing I can be fairly sure of at this point: it's not the server itself.

I live about 1,000 miles from the server, and my netgraph has been a flat green line. I don't recall seeing a single dropped packet yet, and I'm not experiencing lag or jitter.

By a process of elimination, what remains is the "peering" arrangements: the pipe connections between the datacenter and Internet backbone networks.

If you can try a traceroute to tastyspleen.net, it might be interesting to see what route your packets are taking. (Not that we'll have any practical way to improve it.)

That said, my packets seem to be going through gtt.net just like otg's above. So I'm puzzled why he's getting crazy lag and I'm not.

I've also noticed that the admin and quadz avatars don't show up on the forums anymore, haunteds too I believe. Maybe those were loaded into the forums memory and lost during the transitition?

Hmm, thanks for the heads up.  It's throwing a "404 - Attachment Not Found" response. I'll have to look into it (all the files *should* have been copied over, along with the database.)

[otg]

My graph in-game does show many tiny pumps that are random happening. I attached the link to the screenshot.


https://imgur.com/a/5G9yQHE

[|iR|Focalor]

Red spikes have been rampant as a motherfucker today. Ran 2 Tracerts back to back. I edited out my router and static IP address for anti-assholery purposes.

============================================

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\User1>tracert tastyspleen.net

Tracing route to tastyspleen.net [23.227.170.222]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  MyRouter.Home [xxx.xxx.xxx.xxx]
  2     8 ms     7 ms     7 ms  h2.84.134.40.static.ip.windstream.net [xx.xxx.xx
.x]
  3     8 ms     8 ms     8 ms  ae1-0.agr02.cmrc01-ga.us.windstream.net [40.128.
251.102]
  4    15 ms    17 ms    14 ms  ae10-0.cr02.atln02-ga.us.windstream.net [40.132.
58.108]
  5    12 ms    11 ms    11 ms  ae0.cr01.atln02-ga.us.windstream.net [169.130.17
0.166]
  6    12 ms    19 ms    15 ms  173.205.35.17
  7    30 ms    30 ms    30 ms  et-0-0-71.cr11-dal3.ip4.gtt.net [89.149.139.201]

  8    30 ms    30 ms    30 ms  ip4.gtt.net [173.205.47.98]
  9     *        *        *     Request timed out.
 10    31 ms    31 ms    30 ms  Dallas-TX.r19.Public.Pwr-2xPDU-2xUPS-2N-100SLA.i
ncero.com [144.168.34.82]
 11    31 ms    30 ms    31 ms  tastyspleen.net [23.227.170.222]

Trace complete.

C:\Users\User1>tracert tastyspleen.net

Tracing route to tastyspleen.net [23.227.170.222]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  MyRouter.Home [xxx.xxx.xxx.xxx]
  2     8 ms     7 ms     7 ms  h2.84.134.40.static.ip.windstream.net [xx.xxx.xx
.x]
  3     7 ms     7 ms     7 ms  ae1-0.agr02.cmrc01-ga.us.windstream.net [40.128.
251.102]
  4    14 ms    14 ms    14 ms  ae10-0.cr02.atln02-ga.us.windstream.net [40.132.
58.108]
  5    12 ms    11 ms    12 ms  ae0.cr01.atln02-ga.us.windstream.net [169.130.17
0.166]
  6    29 ms    12 ms    12 ms  173.205.35.17
  7    48 ms    30 ms    30 ms  et-0-0-71.cr11-dal3.ip4.gtt.net [89.149.139.201]

  8    30 ms    30 ms    31 ms  ip4.gtt.net [173.205.47.98]
  9     *        *        *     Request timed out.
 10    31 ms    30 ms    31 ms  Dallas-TX.r19.Public.Pwr-2xPDU-2xUPS-2N-100SLA.i
ncero.com [144.168.34.82]
 11    30 ms    30 ms    30 ms  tastyspleen.net [23.227.170.222]

Trace complete.

C:\Users\User1>

============================

#6 seems to be some place in Lawrenceburg Kentucky. No fucking idea why I'd need to hit Kentucky since it should be a straight shot between Atlanta and Dallas. Then again, my info might be wrong. Because #7 has "dal" in it, probably for Dallas, and it's telling me that's in Ireland whilst it's also saying #8 is in Bellevue Washington. Ugh. There's no place to simply put in an IP and find out where the hell it REALLY is anymore is there?

[otg]

I been getting red lines the last couple of days now, this lag has gotten worse as i previously was reporting. Also my ping is up my 10 or so. Can we acknowledge something is going on with the server yet or what???

[Admin]

Can we acknowledge something is going on with the server yet or what???

Only if you wish to promote a narrative that is disconnected from reality.

Which is to say: It's not the server.

The problem is with the route packets are taking to reach the server, which is why the connection can be fine for some players and bad for others.

It's usually been perfect for me, but I've experienced bad packetloss on a couple evenings so far.

As to what, in practical terms, might be done to address this issue: I'm not sure.

(I could envision a modded Q2 server that gathers statistics about which player IPs are experiencing packetloss, and dumping that data out so it could be plotted and graphed over time-of-day, and if a pattern emerged, present the data to the hosting company.)