Data Format FAQ for Passive Traces

Q: I've looked in the Data Cube, but I don't see any raw trace files, where can I get raw header traces?

A: Raw header traces are available on moat.nlanr.net via anonymous ftp or http. Ftp usually presents less problems in the long run, so we suggest that you ftp to moat.nlanr.net/pub/MOAT/Traces/ where we store approximately one week's worth of data before it is backed up and cleared to make room for new data.

Q: I've found the traces I want but how do I decompress the .gz files?

A: You will need to have the gnu zip utility 'gzip' or some other utility that will handle gziped files. Most modern compression utilities will support gzip, including WinZip(tm), and Stuffit Expander(tm).

Q: I have unzipped the .gz files but how do I decompress the .enc file? And what about the .crl file after that?

A: After you have unzipped a file ending in '.crl.enc.gz' you are basically done. Trace headers are made available in the Coral format (.crl) only after the IP addresses have been encoded (.enc) to assure anonymity. This encoding does not affect the resulting statistics in any way. You may wish, however, to use a more manageable format for your purposes. Internally, we primarily use the Time Sequence Header format (.tsh) and there are many tools available (in the form of perl scripts) for converting Coral files to TSH and other useful formats; see our TSH software archive.

Q: I would like to know real IP addresses for my research, is there any way of decoding the .enc files to see real IPs?

A: No. We encode our traces out of concern for the privacy and security of those who use the network. The encoding is a one-way function which ensures that the traces cannot be decoded.

Q: The names of the available traces are confusing, how do I interpret them?

A: Traces are usually named in the form: < Site Abbreviation>-< year,month,day>.[tsh|crl].enc.gz. You can find a table of Site Abbreviations at http://moat.nlanr.net/PMA/Sites.html.

Q: Do I have to give my name and other information in order to access traces?

A: No. The access form is merely for our benefit so we might better know who is using the traces we provide and for what. You may wish to access the traces directly via http or ftp at http://moat.nlanr.net/Traces/Traces.

Q: Where can I find the specific format of _____ Header Trace files?

A: All of the Header Trace file formats are well documented in ascii art at the top of their respective ascii converter scripts. These depictions can also be found on the Header Traces index page at http://moat.nlanr.net/Traces/.

For your convenience, here is the format for the .tsh and .crl formats:

• Time Sequence Header:

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   0  |                    timestamp (seconds)                        | Time
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   1  |  interface #  |          timestamp (microseconds)             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   2  |Version|  IHL  |Type of Service|          Total Length         | IP
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   3  |         Identification        |Flags|      Fragment Offset    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   4  |  Time to Live |    Protocol   |         Header Checksum       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   5  |                       Source Address                          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   6  |                    Destination Address                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   7  |          Source Port          |       Destination Port        | TCP
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   8  |                        Sequence Number                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   9  |                    Acknowledgment Number                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Data |           |U|A|P|R|S|F|                               |
   10 | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
      |       |           |G|K|H|T|N|N|                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  

• Coral Format:

  (packet entries only, not including file header)

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   0  |                          clockstamp                           | Header
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   1  |          clockstamp           |          FIFO depth           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   2  |                          ATM header                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   3  |                           LLC/SNAP                            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   4  |                           LLC/SNAP                            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   5  |Version|  IHL  |Type of Service|          Total Length         | IP
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   6  |         Identification        |Flags|      Fragment Offset    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   7  |  Time to Live |    Protocol   |         Header Checksum       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   8  |                       Source Address                          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   9  |                    Destination Address                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  
    option or other header
  
      |          Source Port          |       Destination Port        | TCP
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                        Sequence Number                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                    Acknowledgment Number                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Data |           |U|A|P|R|S|F|                               |
      | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
      |       |           |G|K|H|T|N|N|                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  

  ---

  
  

• Q: How should the output of the tsh2asc.pl script be interpreted?

  
  A: The tsh2asc.pl script attempts to produce an apt plain text representation of tsh records. One record per line is produced with the following fields:
  1. timestamp, in seconds from the epoch (Jan 1, 1970)
  2. Version/Header length : a two digit hex number.
     The first digit is the Internet Protocol version. The second digit is the IP header length.
  3. Type of Service, in hex.
  4. Total packet length, in bytes.
  5. Source Address
  6. Destination Address
  7. Protocol Number
  8. Source Port
  9. Destination Port

  ---

  
  

  If you have additional questions which are not addressed here, or if you find any of the above information unclear, please feel free to e-mail your comments, suggestions, and questions to dgcheney @ nlanr.net