Loghole - Emulab Log Management Utility

The loghole utility downloads log files from certain directories on the experimental nodes (e.g. "/local/logs") to the Emulab users machine. After downloading, it can also be used to produce and manage archives of the log files.

Using this utility to manage an experiment's log files is encouraged because it will transfer the logs in a network-friendly manner and is already integrated with the rest of Emulab. For example, any programs executed using the Emulab event-system will have their standard output/error automatically placed in the "/local/logs" directory. The tool can also be used to preserve multiple trials of an experiment by producing and managing zip archives of the logs.

You can learn more about the loghole utility by reading its man page on users.emulab.net, included below.



SYNOPSIS

       loghole [-hVdqva] [-e pid/eid] [-s server] [-P port] action [...]

       loghole sync [-nPs] [-r remotedir] [-l localdir] [node1 node2 ...]

       loghole validate

       loghole	archive [-k (i-delete|space-is-needed)] [-a days] [-c comment]
       [-d] [archive-name]

       loghole change [-k (i-delete|space-is-needed)] [-a days]  [-c  comment]
       archive-name1 [archive-name2 ...]

       loghole list [-O1!Xo] [-m atmost] [-s megabytes]

       loghole show [archive-name]

       loghole clean [-fne] [node1 node2 ...]

       loghole gc [-n] [-m atmost] [-s megabytes]


DESCRIPTION

       The loghole utility downloads log files from certain directories on the
       experimental nodes (e.g. "/local/logs") to the  Emulab  users  machine.
       After  downloading,  it can also be used to produce and manage archives
       of the log files.  Using this utility to  manage  an  experiment's  log
       files  is  encouraged  because  it will transfer the logs in a network-
       friendly manner and is already integrated with the rest of Emulab.  For
       example,  any programs executed using the Emulab event-system will have
       their standard output/error automatically placed in  the  "/local/logs"
       directory.  The tool can also be used to preserve multiple trials of an
       experiment by producing and managing zip archives of the logs.

       The set of logs that are actually downloaded  by  the  tool  are  those
       located	in  logholes  on  the nodes, where a loghole is simply a well-
       known directory that acts like a blackhole for log  files.   Any  files
       found  in  these  directories  are  downloaded  to the experiment's log
       directory (i.e. "/proj/<pid>/exp/<eid>/logs") and placed under separate
       directories  for each node and loghole.	The referent of symbolic links
       are also downloaded, so if you do not want an  entire  directory  down-
       loaded, you can create links in a loghole to those files of interest.

       To  perform  its  various  tasks, the loghole utility is broken up into
       several sub-actions that you can apply to an experiment's log holes  or
       log  archives.  As a quick example, to synchronize the logholes for the
       experiment "neptune/myexp" and create an  archive  of  these  logs  you
       would run:

	      [vmars@users ~] loghole -e neptune/myexp sync
	      [vmars@users ~] loghole -e neptune/myexp archive

       The  following  listing	gives  a  brief overview of the current set of

       list   Print  a	brief  listing of the archives in the experiment's log
	      directory.

       show   Print a detailed listing of the archives in the experiment's log
	      directory.

       clean  Clean out the experiment log directory by removing any subdirec-
	      tories and/or clean the log directories on the nodes.

       gc     Garbage collect old archives to free up disk space.

       Options passed to the utility are  split  between  globally  applicable
       ones that are placed before the action and action-specific options that
       are placed after.  For example, the -a  global  option  will  apply  an
       action to all of your experiments and the -X option for the list action
       will list only those files that will be deleted	at  the  next  garbage
       collection.  To combine these options you would enter:

	      [vmars@users ~] loghole -a list -X

       The set of global options is as follows:

       -h, --help
	      Print  the  usage message for the loghole utility as a whole or,
	      if an action is given, the usage message for that action.

       -V, --version
	      Print out version information and exit.

       -d, --debug
	      Output debugging messages.

       -q, --quiet
	      Decrease the level of verbosity, this is subtractive, so	multi-
	      ple  uses  of this option will make the utility quieter and qui-
	      eter.  The default level of verbosity is	human-readable,  below
	      that  is	machine-readable, and below that is silent.  For exam-
	      ple, the default output from the "list" action looks like:

		     [ ] foobar.1.zip	10/15
		     [!] foobar.0.zip	10/13

	      Using a single "-q" option changes the output to look like:

		     foobar.1.zip
		     foobar.0.zip

       -e, --experiment=PID/EID
	      Specify the experiment(s) to operate on.	If multiple -e options
	      are  given,  the	action will apply to all of them.  This option
	      overrides the default behavior of inferring the experiment  from



SYNC

       The  sync  action is used to synchronize the logholes on the nodes with
       the experiment's log directory on the Emulab users machine.  The action
       will  iterate  through  each  node  in  the  experiment	and perform an
       rsync(1) on the loghole directories  for  that  node.   Currently,  the
       default	set  of directories that will be synced are "/var/emulab/logs"
       and "/local/logs".  In addition, if any of the  network	links  in  the
       experiment  are	being traced, loghole will perform the following extra
       steps:

       1.     Create a directory for each link.

       2.     Setup symbolic links to the pcap(3)  files  retrieved  from  the
	      delay nodes.

       3.     Run  the	pre-shaped  pcap(3) files (e.g. *.recv.0) through tcp-
	      trace(1) to generate graphs viewable with xplot(1).   Note  that
	      the processing will only be done for pcap files that were gener-
	      ated by a SNAPSHOT event sent to the tracemon agent.   The  "-s"
	      option  is  provided  to	automatically send this event for each
	      agent.

       While the download is in progress, loghole  will  display  some	simple
       statistics.   The  left-hand  side  of  the display shows the number of
       nodes remaining to be synced, in progress, and completed.   The	right-
       hand  side shows minimum, average, and maximum amount of time needed to
       sync a node.  Finally, a "spinner" on the far right is updated when the
       currently active rsync log files have grown, which usually happens when
       more files are being synced.

       If rsync(1) encounters an error while running, it will automatically be
       rerun after a short delay.

       Optional arguments:

       -r, --remote=remotedir
	      An  additional  remote  directory to sync.  This option is addi-
	      tive, so you can download several additional directories.

       -l, --local=localdir
	      The  local  directory  to  store	the  downloaded  files.   This
	      defaults to the experiment's log directory.

       -n, --no-standard
	      Flag  that indicates that the standard logholes (i.e. "/var/emu-
	      lab/logs", "/local/logs") should not be downloaded.

       -P, --no-post
	      Do not do any additional post-processing of the log files.  Cur-
	      rently,  the only post-processing is done on the pcap files gen-


VALIDATE

       The  validate  action  is  used to check that the logs were sync'd cor-
       rectly.	Currently, the following checks are performed:

       program-agent logs
	      The stdout and stderr logs from program agents  are  checked  by
	      comparing  their metadata against that saved in the accompanying
	      ".status" files.

       valid soft links
	      All soft links are checked to ensure the referent exists.


ARCHIVE

       The archive action is used to archive the logs in an  experiment's  log
       directory for future reference.	The action will produce a standard zip
       archive with the logs and some metadata about the experiment  and  when
       it can be garbage collected.

       Available options:

       -k, --keep-until=(i-delete|space-is-needed)
	      Keep the archive until you decide to delete it manually or space
	      is needed.  See the GC section later in the manual to learn  how
	      this  option  and  others  affect garbage collection.  (Default:
	      space-is-needed)

       -a, --keep-atleast=N
	      Keep the archive atleast N  days	after  creation.   This  value
	      keeps  the  archive from being garbage collected when more space
	      is needed for atleast the given number  of  days.   (Default:  3
	      days)

       -c, --comment=COMMENT
	      Add  a comment to the archive.  This option can be used multiple
	      times to add more than one comment to the archive.  The comments
	      will be displayed by the show action and can be useful for stor-
	      ing information about the experiment,  for  example,  the  input
	      parameters.   If the argument to this option is is a single dash
	      (-) the comment will be read from standard in.

       -d, --delete
	      After creating the archive, mark the experiment as clean-on-sync
	      so that any stale data is cleaned out before performing the next
	      sync operation.  Currently, the mark is a file named  ".cleanon-
	      sync" in the experiment log directory.


CHANGE

       The  change  action  is	used to change the metadata of an existing ar-
       chive.  For example, if after analyzing the log files, you decide  that
       they represent "good" data, you can add a comment stating that fact and
       mark the archive as not garbage collectable.  The action takes the same
       set of options as the archive action.
       -X     Only list archives that are ready to be garbage collected.

       -o     List archives that do not  match	the  above  flags.   In  other
	      words,  archives	that  will  not be deleted at the next garbage
	      collection and are more than a day away from their  keep-atleast
	      dates.

       -m, --keep-atmost=N
	      Specify  how  many  archives  should  be kept in the experiment.
	      This setting effects what files will be  garbage	collected,  so
	      you  should pass this same option to the gc if you use a differ-
	      ent value from the default of 100 archives.

       -s, --keep-size=megabytes
	      Specify the maximum total size, in megabytes, for all of the ar-
	      chives  in the experiment.  This setting effects what files will
	      be garbage collected, so you pass this same option to the gc  if
	      you use a different value from the default of 3MB.


SHOW

       The  show  action  provides a more detailed listing of the log archives
       for an experiment.  The listing contains information about when and who
       created	the  archive, any attributes used when computing the GC status
       of the archive, comments attached to the archive, and a listing of  the
       files in the archive.

       Optional arguments:

       archive-name
	      The  full  or partial name of the archive to display.  If a par-
	      tial name is given, any archive names that start with the  argu-
	      ment  are  displayed.  The default behavior is to display all of
	      the archives in an experiment.


CLEAN

       The clean action is used to clean out log files from  the  experiment's
       log directory and the log directories on the nodes.  The default action
       is to ask confirmation and then clean out all of the log files.

       Available clean options:

       -f, --force
	      Do not prompt for confirmation.

       -n, --nodes
	      Only remove log files on the nodes.

       -e, --experiment
	      Only remove log directories in the experiment's log directory.

       -R, --root
	      Use sudo to run the clean as root on the nodes

       3.     The  oldest files that are marked 'keep until "space-is-needed"'
	      will be deleted until the keep-atmost and  keep-size  conditions
	      are  met	or there are no more files that can be deleted without
	      user intervention.

       Available gc options:

       -m, --keep-atmost=N
	      Specify how many archives should	be  kept  in  the  experiment.
	      (Default: 100 archives)

       -s, --keep-size=megabytes
	      Specify the maximum total size, in megabytes, for all of the ar-
	      chives in the experiment.  (Default: 3.0 MB)


ENVIRONMENT

       By default, the project and experiment ID will  be  inferred  from  the
       current	working  directory, if it is inside the experiment's directory
       (i.e. /proj/pid/exp/eid).  This behavior can be overridden using the -e
       option.


RETURN VALUES

       3      If rsync reports an error.

       2      If there was an error processing the command line arguments.

       0      If the action was completed successfully.


EXAMPLES

       To  synchronize	the  log directory for experiment "neptune/myexp" with
       the log holes on the experimental nodes.

	      [vmars@users ~] loghole -e neptune/myexp sync

       To archive the newly recovered logs and print out just the name of  the
       new log file:

	      [vmars@users ~] loghole -e neptune/myexp -q archive


FILES

       /proj/pid/exp/eid/logs
	      The log directory for an experiment.

       /proj/pid/exp/eid/logs/node
	      The log directory for a node in the experiment.

       /proj/pid/exp/eid/logs/link
	      The  log directory for a traced LAN or link.  The symbolic links
	      in these directories refer to the pcap(3) files for a node  con-
	      nected to this LAN or link.

	      cally  sync'd.   This  directory usually holds logs generated by
	      the Emulab software running on the node.


SEE ALSO

       event-sched(8), tevc(1), zip(1), rsync(1), pcap(3),  mergecap(1),  tcp-
       trace(1), xplot(1)


AUTHOR

       The Emulab project at the University of Utah.


NOTES

       The Emulab project can be found on the web at http://www.emulab.net



Emulab				 June 16, 2005			    LOGHOLE(1)