Introducing d-SCI

[kren.Z]

The Digital Scientific and Surveillance System.




About: Advanced Network Diagnostic and Analysis Tool.
Prototype: http://krenz.tastyspleen.net/geolocate/



Ok, so this a very simple prototype and I'm still working on implementing a whole host of functionality, but you can see what direction I'm heading with this program.
Google recently launched mlabs [http://www.measurementlab.net/] that will provide a similar service, but in a much different way. My reason for creating dsci is to  provide a visual interface to the diagnostic data that mlabs produces. So what i need are sources and material on sophisticated networking algorithms that generally provide routing statistics, ISP information, traffic loads etc etc. Also, if you guys have any feedback that will help to (bugs and such).


 - a big thanks goes to quadz for his guidance and genius.

[reaper]

I don't mean to knock your work, but I will add my perspective.  I do not see any usefullness in mlabs.  A simple test with iperf ( http://en.wikipedia.org/wiki/Iperf ) or a myriad of other tools will do just what mlabs is trying to accomplish.

So you run iperf on two computers, and it fills the pipe as much as possible  by changing the window size (uses an extended option, just shifts the bit, to make the window size huge), or use udp.  So that will max out the connection to it's full potential, to see if things are functioning as designed.

My only guess is this is some type of aggregate system for internet health, or provider service level measurements across a large user base, to review providers.  Which as far as I'm concerned has almost no use, and has already been done.

As far as sophisitcated networking algortihms for gathering traffic statistics, BGP table sizes etc, this is avaialble to the provider through snmp.  It's a simple use of the snmp protocol, you request a get for a certain object, and you get back the info. like..
get interface.e0.out.total.traffic, and you get the traffic stats.  There is also Netflow where the router exports statistics, like IP, port, bytes out , etc. so you see information at the tcp, and IP level (what's using the pipe).  This can be created by a server as well, but routers do it a little cleanly..just a command or two, and it's done.  To look into the data more , you need packet capture software, or some type of device that understands what in the packets.  For example, a tcp connection threshold is reached for an IP, so you send back a tcp/rst to kill the connection.  Or even some type of algorithm to throttle data, or drop connections, if the IP is spoofed, for things like syn floods.  Sometimes the firewall proxies the connection, and is designed to handle the memory allocation for these half open connections fine.  Anyways, I don't see any shortcoming in just iperf itself, it will test a connection just fine.

The alogrithms that are complicated are usually in the routing protocol itself to choose the best path (route).  Which are based on math, http://en.wikipedia.org/wiki/Dijkstra%27s_algorithm , really it doesn't even need to be that complicated though, there are routing protocols based on simple metrics like hop count that scale upwards of 5,000+ routes, and you can even link internal networks when they get big through BGP.  Although OSPF which uses the dijkstra alogrithm is usually considered the standard choice.

Anyways that's just my opinion that mlabs doesn't appear to be doing anything worthwhile.  What I do see as worthwhile are the NOC's at Time Warner and Akamai that actually fix the problem when things like the slammer worm come about.

edit: I want to add this as well. To setup netflow on a server for your connection, mirror a port that is your connection (send all your internet traffic to another port on a server), and use fprobe, flow-scan,and CUgrapher.pl (you can find them on google).  It's a nice perl script that reads the netflow data fprobe creates of the mirrored port (flow-scan collects the netflow data).  I've done it, and imo it's a bit of a pain, because of all the switches you need to set between the script and fprobe, and flow-scan.  But it does work.

The result will be a web app, that bytes out per ip, per port etc.  And you can quickly see the top talkers.

[kren.Z]

reaper....










THANK YOU.

This is exactly the kind of feed back I wanted. I was hoping that you would respond since I knew that you were a networking guy. Thanks again.

[reaper]

no problem

I like the globe

are you going to have a database of source connections to search through?  I'm not sure if you have a source network to a destination (tastyspleen), how accurate it would be.  It depends if the path is the same, and to what extent.  The source network might be a good indication, but sometimes it will not be the same if you don't get the same path, it could be bad.

You'd also have to have this on tons of clients, and you'd need data to tastyspleen.  What are you going to do with the program?  It didn't load in my browser or something before my last post.  Right now I would do whois, and ping, there is visual traceroute to, which looks kind of nice, if you're looking for a visual effect.

[[BTF] Reflex]

was looking for something in china or japan to test.. 

[kren.Z]

Implementing whois and trace route as we speak.



Reaper,
I have a much grander vision for the d-sci system itself, but for now I'm just starting small. I want to provide some useful tools for users to troubleshoot their connections to the tastyspleen servers. (of course, in a much more elicit fashion).


However, I'm running into some limitations when outlining some extended functionality to platform. For example, how to map the data route from source terminal to the destination terminal without tastyspleen acting as a proxy. (and i really don't want to make this a stand-alone unless i really have to). Any ideas?


I really appreciate the advice and guidance reap. You threw a lot at me and I'm just trying to digest of how I'm going  to program these back-end tools. If you have any other tips I'm all ears. 


I've been optimizing the hell out of this thing. I've debated as to whether or not to implement a pre-loader but I've managed to cut down so much file size that I just didn't need it (or so i thought). I'm curious as to why it didn't load for you. Other than that, how would you rate the performance?


Reflex,
thanks for giving it a spin. It seems that my geolocation data is holding up fairly well, but it seems that the most detailed information is restricted to those systems in the United States.


-krenZ

[reaper]

Ideally you will want the client to initiate the traceroute. 

The path may not be the same from source to destination as destination to source.  While you measure round trip time from the server , you are not getting any indication of what hop is slow from the client to server.  While pathping is good, a better aproach might be to ping through the router somehow (locate an attached network, and live host), because routers have a data-plane (packet routed in ASIC), and a forwarding-plane (packet treateD/routed by CPU). The packet with an expiered TTL value are handled by the control-plane.  Anyways, I'm not sure if pathping usually suffices, but it certainly does indicate what hop is congested most of the time, so maybe you want to start with pathping and traceroute.

You could push an ActiveX control to the client that uses traceroute and pathping (or you're own version of the tool without calling to a system command), you'd want to register it to get the least problems with browsers.  I'm not sure what else through the browser could control the client and work with system commands, or open sockets, but some technologies will, flash, maybe.

[opt1ckz]

The Digital Scientific and Surveillance System.




About: Advanced Network Diagnostic and Analysis Tool.
Prototype: http://krenz.tastyspleen.net/geolocate/



Ok, so this a very simple prototype and I'm still working on implementing a whole host of functionality, but you can see what direction I'm heading with this program.
Google recently launched mlabs [http://www.measurementlab.net/] that will provide a similar service, but in a much different way. My reason for creating dsci is to  provide a visual interface to the diagnostic data that mlabs produces. So what i need are sources and material on sophisticated networking algorithms that generally provide routing statistics, ISP information, traffic loads etc etc. Also, if you guys have any feedback that will help to (bugs and such).


 - a big thanks goes to quadz for his guidance and genius.

you need to clear the buffer after the ping and whois scripts. also, direct the from the REMOTE computer that is connecting to your site to whois or ping certain hosts. it returns pings from the LOCAL computer to the host. :\ i'm sure it's not that hard to change.

[kren.Z]

sup k_,

you need to clear the buffer after the ping and whois scripts. also, direct the from the REMOTE computer that is connecting to your site to whois or ping certain hosts. it returns pings from the LOCAL computer to the host. :\ i'm sure it's not that hard to change.

Can you elaborate on this? I really don't understand what you're trying to elucidate. The tools implemented have no need for requiring a direct data route from local machine to the remote host. Ping is just a simple network tool to check whether a machine is reachable via a network. However, I am running into considerable difficulties extending the functionality for more sophisticated analysis.

[reaper]

"the tools implemented have no need for requiring a direct data route from the local machine"

what are you trying to do?

wouldn't it be nice to see the ping from where your connecting, and not the server?