From: Joerg Micheel (joerg@cs.waikato.ac.nz)
Date: Mon Jan 22 2001 - 15:24:57 PST
On Mon, Jan 22, 2001 at 01:35:03PM -0800, Bo Ryu wrote:
> From this discussion, it seems clear that we need to find a way to get a
> long-duration trace or extend the capability of the current monitors to
> achieve this without needing humongous disk storage space. I shall be
> grateful if someone can tell me whether it is possible to "pipeline" the
> basic data (packet timestamp and header information) on-the-fly into other
> "engines" that process this data for whatever purpose we have (e.g.,
> protocol mix or wavelet analysis) in real time, and delete this basic data
> after they are fed into these engines to recycle the storage space.
Technically, it is possible. The NZIX-II data set has been produced that
way. We have captured the data, run it via a pipe into the anonymizer
(which keeps a database of IP address mappings), into gzip, across a
10 MBit/sec switched network, stored onto our main data server. Of
course, this is kind-of crazy, but was the only option available.
For higher data rates, a different approach will be required. What's
clear to me is that the current perl-based analysis is on the way out.
If computation is feasible, it must be C - hardcoded. I think we can
gain quite a few cycles by reimplementing our tools.
Whether it is possible to run *your* code on any of the monitors is
something we have not considered so far. There is this security concern
that we need to check over and over again - which kind of data you can
actually remove from the monitors without breaching into anyones security
and privacy. I think, for the time being we will consider traces, as it
is of benefit to a large audience, also encourages comparative analysis
of different groups on the same data set. All of the work can thus be
verified by everyone else.
An option that is always available is to install your own monitor at
a link that you have access to, work with the raw traces and compare
your results with data captured from the NLANR infrastructure. This
is, in fact, what the people at UNC-CH did.
Joerg
-- Joerg B. Micheel Email: <joerg@cs.waikato.ac.nz> WAND and NLANR MOAT Email: <joerg@nlanr.net> The University of Waikato, CompScience Phone: +64 7 8384794 Private Bag 3105 Fax: +64 7 8585095 Hamilton, New Zealand Plan: PMA, TINE and the DAG's
This archive was generated by hypermail 2b30 : Thu Sep 27 2001 - 16:24:41 PDT