Author Topic: Introducing d-SCI  (Read 2739 times)

kren.Z

  • Guest
Introducing d-SCI
« on: February 03, 2009, 06:21:55 PM »

The Digital Scientific and Surveillance System.




About: Advanced Network Diagnostic and Analysis Tool.
Prototype: http://krenz.tastyspleen.net/geolocate/



Ok, so this a very simple prototype and I'm still working on implementing a whole host of functionality, but you can see what direction I'm heading with this program.
Google recently launched mlabs [http://www.measurementlab.net/] that will provide a similar service, but in a much different way. My reason for creating dsci is to  provide a visual interface to the diagnostic data that mlabs produces. So what i need are sources and material on sophisticated networking algorithms that generally provide routing statistics, ISP information, traffic loads etc etc. Also, if you guys have any feedback that will help to (bugs and such).


 - a big thanks goes to quadz for his guidance and genius.

« Last Edit: January 28, 2014, 01:50:29 PM by krenZ »
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

Offline reaper

  • Opulent Member
  • *
  • Posts: 2872
  • Nice night for a walk, eh? - Nice night for a walk
    • View Profile
  • Rated:
Re: Introducing d-SCI
« Reply #1 on: February 03, 2009, 08:28:00 PM »
I don't mean to knock your work, but I will add my perspective.  I do not see any usefullness in mlabs.  A simple test with iperf ( http://en.wikipedia.org/wiki/Iperf ) or a myriad of other tools will do just what mlabs is trying to accomplish.

So you run iperf on two computers, and it fills the pipe as much as possible  by changing the window size (uses an extended option, just shifts the bit, to make the window size huge), or use udp.  So that will max out the connection to it's full potential, to see if things are functioning as designed.

My only guess is this is some type of aggregate system for internet health, or provider service level measurements across a large user base, to review providers.  Which as far as I'm concerned has almost no use, and has already been done.

As far as sophisitcated networking algortihms for gathering traffic statistics, BGP table sizes etc, this is avaialble to the provider through snmp.  It's a simple use of the snmp protocol, you request a get for a certain object, and you get back the info. like..
get interface.e0.out.total.traffic, and you get the traffic stats.  There is also Netflow where the router exports statistics, like IP, port, bytes out , etc. so you see information at the tcp, and IP level (what's using the pipe).  This can be created by a server as well, but routers do it a little cleanly..just a command or two, and it's done.  To look into the data more , you need packet capture software, or some type of device that understands what in the packets.  For example, a tcp connection threshold is reached for an IP, so you send back a tcp/rst to kill the connection.  Or even some type of algorithm to throttle data, or drop connections, if the IP is spoofed, for things like syn floods.  Sometimes the firewall proxies the connection, and is designed to handle the memory allocation for these half open connections fine.  Anyways, I don't see any shortcoming in just iperf itself, it will test a connection just fine.

The alogrithms that are complicated are usually in the routing protocol itself to choose the best path (route).  Which are based on math, http://en.wikipedia.org/wiki/Dijkstra%27s_algorithm , really it doesn't even need to be that complicated though, there are routing protocols based on simple metrics like hop count that scale upwards of 5,000+ routes, and you can even link internal networks when they get big through BGP.  Although OSPF which uses the dijkstra alogrithm is usually considered the standard choice.

Anyways that's just my opinion that mlabs doesn't appear to be doing anything worthwhile.  What I do see as worthwhile are the NOC's at Time Warner and Akamai that actually fix the problem when things like the slammer worm come about.

edit: I want to add this as well. To setup netflow on a server for your connection, mirror a port that is your connection (send all your internet traffic to another port on a server), and use fprobe, flow-scan,and CUgrapher.pl (you can find them on google).  It's a nice perl script that reads the netflow data fprobe creates of the mirrored port (flow-scan collects the netflow data).  I've done it, and imo it's a bit of a pain, because of all the switches you need to set between the script and fprobe, and flow-scan.  But it does work.

The result will be a web app, that bytes out per ip, per port etc.  And you can quickly see the top talkers.
« Last Edit: February 03, 2009, 08:46:05 PM by reaper »
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus
VaeVictus "reaper is a lying sack of shit and ragequit then had, probably slugs, come alias and beat me, wasnt even the same person playing OBVIOUSLY, accuracies basicly doubled, and strategy

kren.Z

  • Guest
Re: Introducing d-SCI
« Reply #2 on: February 03, 2009, 08:38:15 PM »
reaper....










THANK YOU.

This is exactly the kind of feed back I wanted. I was hoping that you would respond since I knew that you were a networking guy. Thanks again.
« Last Edit: January 28, 2014, 01:50:31 PM by krenZ »
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

Offline reaper

  • Opulent Member
  • *
  • Posts: 2872
  • Nice night for a walk, eh? - Nice night for a walk
    • View Profile
  • Rated:
Re: Introducing d-SCI
« Reply #3 on: February 04, 2009, 09:24:04 PM »
no problem

I like the globe

are you going to have a database of source connections to search through?  I'm not sure if you have a source network to a destination (tastyspleen), how accurate it would be.  It depends if the path is the same, and to what extent.  The source network might be a good indication, but sometimes it will not be the same if you don't get the same path, it could be bad.

You'd also have to have this on tons of clients, and you'd need data to tastyspleen.  What are you going to do with the program?  It didn't load in my browser or something before my last post.  Right now I would do whois, and ping, there is visual traceroute to, which looks kind of nice, if you're looking for a visual effect.


  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus
VaeVictus "reaper is a lying sack of shit and ragequit then had, probably slugs, come alias and beat me, wasnt even the same person playing OBVIOUSLY, accuracies basicly doubled, and strategy

Offline [BTF] Reflex

  • Loquaciously Multiloquent Member
  • ****
  • Posts: 5291
  • !
    • View Profile
  • Rated:
Re: Introducing d-SCI
« Reply #4 on: February 06, 2009, 05:52:13 AM »
was looking for something in china or japan to test.. 
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus
Sometimes I think it’s a sin when I feel like I’m winnin’ when I’m losin’ again

kren.Z

  • Guest
Re: Introducing d-SCI
« Reply #5 on: February 06, 2009, 10:13:16 AM »
Implementing whois and trace route as we speak.



Reaper,
I have a much grander vision for the d-sci system itself, but for now I'm just starting small. I want to provide some useful tools for users to troubleshoot their connections to the tastyspleen servers. (of course, in a much more elicit fashion).


However, I'm running into some limitations when outlining some extended functionality to platform. For example, how to map the data route from source terminal to the destination terminal without tastyspleen acting as a proxy. (and i really don't want to make this a stand-alone unless i really have to). Any ideas?


I really appreciate the advice and guidance reap. You threw a lot at me and I'm just trying to digest of how I'm going  to program these back-end tools. If you have any other tips I'm all ears.  ;D


I've been optimizing the hell out of this thing. I've debated as to whether or not to implement a pre-loader but I've managed to cut down so much file size that I just didn't need it (or so i thought). I'm curious as to why it didn't load for you. Other than that, how would you rate the performance?


Reflex,
thanks for giving it a spin. It seems that my geolocation data is holding up fairly well, but it seems that the most detailed information is restricted to those systems in the United States.


-krenZ
« Last Edit: January 28, 2014, 01:50:32 PM by krenZ »
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

Offline reaper

  • Opulent Member
  • *
  • Posts: 2872
  • Nice night for a walk, eh? - Nice night for a walk
    • View Profile
  • Rated:
Re: Introducing d-SCI
« Reply #6 on: February 06, 2009, 11:48:35 AM »
Ideally you will want the client to initiate the traceroute. 

The path may not be the same from source to destination as destination to source.  While you measure round trip time from the server , you are not getting any indication of what hop is slow from the client to server.  While pathping is good, a better aproach might be to ping through the router somehow (locate an attached network, and live host), because routers have a data-plane (packet routed in ASIC), and a forwarding-plane (packet treateD/routed by CPU). The packet with an expiered TTL value are handled by the control-plane.  Anyways, I'm not sure if pathping usually suffices, but it certainly does indicate what hop is congested most of the time, so maybe you want to start with pathping and traceroute.

You could push an ActiveX control to the client that uses traceroute and pathping (or you're own version of the tool without calling to a system command), you'd want to register it to get the least problems with browsers.  I'm not sure what else through the browser could control the client and work with system commands, or open sockets, but some technologies will, flash, maybe.
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus
VaeVictus "reaper is a lying sack of shit and ragequit then had, probably slugs, come alias and beat me, wasnt even the same person playing OBVIOUSLY, accuracies basicly doubled, and strategy

Offline opt1ckz

  • Jr. Member
  • **
  • Posts: 82
    • View Profile
  • Rated:
Re: Introducing d-SCI
« Reply #7 on: February 20, 2009, 10:45:30 AM »

The Digital Scientific and Surveillance System.




About: Advanced Network Diagnostic and Analysis Tool.
Prototype: http://krenz.tastyspleen.net/geolocate/



Ok, so this a very simple prototype and I'm still working on implementing a whole host of functionality, but you can see what direction I'm heading with this program.
Google recently launched mlabs [http://www.measurementlab.net/] that will provide a similar service, but in a much different way. My reason for creating dsci is to  provide a visual interface to the diagnostic data that mlabs produces. So what i need are sources and material on sophisticated networking algorithms that generally provide routing statistics, ISP information, traffic loads etc etc. Also, if you guys have any feedback that will help to (bugs and such).


 - a big thanks goes to quadz for his guidance and genius.


you need to clear the buffer after the ping and whois scripts. also, direct the from the REMOTE computer that is connecting to your site to whois or ping certain hosts. it returns pings from the LOCAL computer to the host. :\ i'm sure it's not that hard to change.
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

kren.Z

  • Guest
Re: Introducing d-SCI
« Reply #8 on: February 21, 2009, 02:29:05 PM »
sup k_,

you need to clear the buffer after the ping and whois scripts. also, direct the from the REMOTE computer that is connecting to your site to whois or ping certain hosts. it returns pings from the LOCAL computer to the host. :\ i'm sure it's not that hard to change.

Can you elaborate on this? I really don't understand what you're trying to elucidate. The tools implemented have no need for requiring a direct data route from local machine to the remote host. Ping is just a simple network tool to check whether a machine is reachable via a network. However, I am running into considerable difficulties extending the functionality for more sophisticated analysis.
« Last Edit: January 28, 2014, 01:50:34 PM by krenZ »
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus

Offline reaper

  • Opulent Member
  • *
  • Posts: 2872
  • Nice night for a walk, eh? - Nice night for a walk
    • View Profile
  • Rated:
Re: Introducing d-SCI
« Reply #9 on: February 27, 2009, 12:36:23 PM »
"the tools implemented have no need for requiring a direct data route from the local machine"

what are you trying to do?

wouldn't it be nice to see the ping from where your connecting, and not the server?
  • Insightful
    Informative
    Funny
    Nice Job / Good Work
    Rock On
    Flawless Logic
    Well-Reasoned Argument and/or Conclusion
    Demonstrates Exceptional Knowlege of the Game
    Appears Not to Comprehend Game Fundamentals
    Frag of the Week
    Frag Hall of Fame
    Jump of the Week
    Jump Hall of Fame
    Best Solution
    Wins The Internet
    Whoosh! You done missed the joke thar Cletus!
    Obvious Troll Is Obvious
    DO YOU EVEN LIFT?
    DEMO OR STFU
    Offtopic
    Flamebait
    Redundant
    Factually Challenged
    Preposterously Irrational Arguments
    Blindingly Obvious Logical Fallacies
    Absurd Misconstrual of Scientific Principles or Evidence
    Amazing Conspiracy Theory Bro
    Racist Ignoramus
VaeVictus "reaper is a lying sack of shit and ragequit then had, probably slugs, come alias and beat me, wasnt even the same person playing OBVIOUSLY, accuracies basicly doubled, and strategy

 

El Box de Shoutamente

Last 10 Shouts:

 

Admin

August 03, 2020, 11:49:52 PM
THIS SUNDAY !

9 AM PACIFIC / 12 PM EASTERN / 5 PM CET
WINNER IS FIRST TO 500 FRAGS
(GOTO TS500)
 

NoBugsOnMe

August 03, 2020, 11:10:39 PM

Punk_FAS

August 03, 2020, 11:58:12 AM
Nice. I didn't know there was such a thread. Thanks :)
 

quadz

August 03, 2020, 11:37:31 AM
Shoutbox isn't the place for rehashing this. Here's the thread about DM server spawn farthest from 2006/2007:

http://forum.tastyspleen.net/quake/index.php?topic=2932.0

Punk_FAS

August 03, 2020, 11:21:37 AM
but that's *NOT what the DM server is supposed to cater to

Punk_FAS

August 03, 2020, 11:21:09 AM
It's not ideal for 1v1 situations, but that's what the DM server is supposed to cater to. Also, there are 7 other maps besides DM4 (actually 8 for the automatic mymap map between dm8 and dm1). I think there are more positives than negatives.

Punk_FAS

August 03, 2020, 11:20:06 AM
It happens to better players too. I know I've been in first, or second place, only to die and spawn right next to the other leading player multiple times in a row, and having to dodge rockets in a narrow corridor without any weapons myself (other than blaster).

It's not the best for 1v1 situation

Punk_FAS

August 03, 2020, 11:17:56 AM
I saw a clearly-new player earlier, and it spawned him like 3 or 4 times in a row right next to a better player. Instant death each time they spawned. I imagine that can be discouraging and frustrating, particularly when someone is trying to get a handle on the game. Spawn farthest would avoid this.
 

quadz

August 03, 2020, 10:29:41 AM
Might be interesting to code a hybrid spawn function that doesn't focus on "farthest" so much as avoiding the few most congested spawn points?
 

quadz

August 03, 2020, 10:26:44 AM
We'd tried spawn farthest many moons ago, but it seemed a net negative overall. At least for larger maps like q2dm4, where one tends to get spawned furthest from all the action.

In any case, we have a 1-second spawn protect shield, and it has been suggested if THAT isn't good enough, one might consider transitioning to Minesweeper or Candy Crush. ;)

Show 50 latest
Welcome, Guest. Please login or register.
August 04, 2020, 10:19:52 AM

Login with username, password and session length