From: Bo Ryu (ryu@hrl.com)
Date: Mon Jan 22 2001 - 16:02:24 PST
Joerg,
> On Mon, Jan 22, 2001 at 01:35:03PM -0800, Bo Ryu wrote:
> > From this discussion, it seems clear that we need to find a way to get a
> > long-duration trace or extend the capability of the current monitors to
> > achieve this without needing humongous disk storage space. I shall be
> > grateful if someone can tell me whether it is possible to "pipeline" the
> > basic data (packet timestamp and header information) on-the-fly
> into other
> > "engines" that process this data for whatever purpose we have (e.g.,
> > protocol mix or wavelet analysis) in real time, and delete this
> basic data
> > after they are fed into these engines to recycle the storage space.
>
> Technically, it is possible. The NZIX-II data set has been produced that
> way. We have captured the data, run it via a pipe into the anonymizer
> (which keeps a database of IP address mappings), into gzip, across a
> 10 MBit/sec switched network, stored onto our main data server. Of
> course, this is kind-of crazy, but was the only option available.
> For higher data rates, a different approach will be required. What's
> clear to me is that the current perl-based analysis is on the way out.
> If computation is feasible, it must be C - hardcoded. I think we can
> gain quite a few cycles by reimplementing our tools.
I am encouraged to hear that it is feasible to insert our own analysis tools
in C. All the programs we have been developing are in C, and have been
optimized for computational efficiency. I hope to have more discussion along
this avenue in the near future after we try them using already collected
long traces.
>
> Whether it is possible to run *your* code on any of the monitors is
> something we have not considered so far. There is this security concern
> that we need to check over and over again - which kind of data you can
> actually remove from the monitors without breaching into anyones security
> and privacy. I think, for the time being we will consider traces, as it
> is of benefit to a large audience, also encourages comparative analysis
> of different groups on the same data set. All of the work can thus be
> verified by everyone else.
I fully understand the security implications. If our code can be inserted,
they will be dealing with sanitized traces to begin with, and we will only
work with packet timestamp (and perhaps size) for real-time wavelet
analysis. This should be straightfoward to check in the C code.
>
> An option that is always available is to install your own monitor at
> a link that you have access to, work with the raw traces and compare
> your results with data captured from the NLANR infrastructure. This
> is, in fact, what the people at UNC-CH did.
Yes, we are in the plan to try that in our network as well. But our access
rate is only T1 and not much useful from research perspective other than
tool validation.
We will keep you updated.
-Bo
>
> Joerg
> --
> Joerg B. Micheel Email: <joerg@cs.waikato.ac.nz>
> WAND and NLANR MOAT Email: <joerg@nlanr.net>
> The University of Waikato, CompScience Phone: +64 7 8384794
> Private Bag 3105 Fax: +64 7 8585095
> Hamilton, New Zealand Plan: PMA, TINE and the DAG's
This archive was generated by hypermail 2b30 : Thu Sep 27 2001 - 16:24:41 PDT