Using this utility to manage an experiment's log files is encouraged because it will transfer the logs in a network-friendly manner and is already integrated with the rest of Emulab. For example, any programs executed using the Emulab event-system will have their standard output/error automatically placed in the "/local/logs" directory. The tool can also be used to preserve multiple trials of an experiment by producing and managing zip archives of the logs.
You can learn more about the loghole utility by reading its man page on users.emulab.net, included below.
loghole [-hVdqva] [-e pid/eid] [-s server] [-P port] action [...]
loghole sync [-nPs] [-r remotedir] [-l localdir] [node1 node2 ...]
loghole validate
loghole archive [-k (i-delete|space-is-needed)] [-a days] [-c comment]
[-d] [archive-name]
loghole change [-k (i-delete|space-is-needed)] [-a days] [-c comment]
archive-name1 [archive-name2 ...]
loghole list [-O1!Xo] [-m atmost] [-s megabytes]
loghole show [archive-name]
loghole clean [-fne] [node1 node2 ...]
loghole gc [-n] [-m atmost] [-s megabytes]
The loghole utility downloads log files from certain directories on the
experimental nodes (e.g. "/local/logs") to the Emulab users machine.
After downloading, it can also be used to produce and manage archives
of the log files. Using this utility to manage an experiment's log
files is encouraged because it will transfer the logs in a network-
friendly manner and is already integrated with the rest of Emulab. For
example, any programs executed using the Emulab event-system will have
their standard output/error automatically placed in the "/local/logs"
directory. The tool can also be used to preserve multiple trials of an
experiment by producing and managing zip archives of the logs.
The set of logs that are actually downloaded by the tool are those
located in logholes on the nodes, where a loghole is simply a well-
known directory that acts like a blackhole for log files. Any files
found in these directories are downloaded to the experiment's log
directory (i.e. "/proj/<pid>/exp/<eid>/logs") and placed under separate
directories for each node and loghole. The referent of symbolic links
are also downloaded, so if you do not want an entire directory down-
loaded, you can create links in a loghole to those files of interest.
To perform its various tasks, the loghole utility is broken up into
several sub-actions that you can apply to an experiment's log holes or
log archives. As a quick example, to synchronize the logholes for the
experiment "neptune/myexp" and create an archive of these logs you
would run:
[vmars@users ~] loghole -e neptune/myexp sync
[vmars@users ~] loghole -e neptune/myexp archive
The following listing gives a brief overview of the current set of
list Print a brief listing of the archives in the experiment's log
directory.
show Print a detailed listing of the archives in the experiment's log
directory.
clean Clean out the experiment log directory by removing any subdirec-
tories and/or clean the log directories on the nodes.
gc Garbage collect old archives to free up disk space.
Options passed to the utility are split between globally applicable
ones that are placed before the action and action-specific options that
are placed after. For example, the -a global option will apply an
action to all of your experiments and the -X option for the list action
will list only those files that will be deleted at the next garbage
collection. To combine these options you would enter:
[vmars@users ~] loghole -a list -X
The set of global options is as follows:
-h, --help
Print the usage message for the loghole utility as a whole or,
if an action is given, the usage message for that action.
-V, --version
Print out version information and exit.
-d, --debug
Output debugging messages.
-q, --quiet
Decrease the level of verbosity, this is subtractive, so multi-
ple uses of this option will make the utility quieter and qui-
eter. The default level of verbosity is human-readable, below
that is machine-readable, and below that is silent. For exam-
ple, the default output from the "list" action looks like:
[ ] foobar.1.zip 10/15
[!] foobar.0.zip 10/13
Using a single "-q" option changes the output to look like:
foobar.1.zip
foobar.0.zip
-e, --experiment=PID/EID
Specify the experiment(s) to operate on. If multiple -e options
are given, the action will apply to all of them. This option
overrides the default behavior of inferring the experiment from
The sync action is used to synchronize the logholes on the nodes with
the experiment's log directory on the Emulab users machine. The action
will iterate through each node in the experiment and perform an
rsync(1) on the loghole directories for that node. Currently, the
default set of directories that will be synced are "/var/emulab/logs"
and "/local/logs". In addition, if any of the network links in the
experiment are being traced, loghole will perform the following extra
steps:
1. Create a directory for each link.
2. Setup symbolic links to the pcap(3) files retrieved from the
delay nodes.
3. Run the pre-shaped pcap(3) files (e.g. *.recv.0) through tcp-
trace(1) to generate graphs viewable with xplot(1). Note that
the processing will only be done for pcap files that were gener-
ated by a SNAPSHOT event sent to the tracemon agent. The "-s"
option is provided to automatically send this event for each
agent.
While the download is in progress, loghole will display some simple
statistics. The left-hand side of the display shows the number of
nodes remaining to be synced, in progress, and completed. The right-
hand side shows minimum, average, and maximum amount of time needed to
sync a node. Finally, a "spinner" on the far right is updated when the
currently active rsync log files have grown, which usually happens when
more files are being synced.
If rsync(1) encounters an error while running, it will automatically be
rerun after a short delay.
Optional arguments:
-r, --remote=remotedir
An additional remote directory to sync. This option is addi-
tive, so you can download several additional directories.
-l, --local=localdir
The local directory to store the downloaded files. This
defaults to the experiment's log directory.
-n, --no-standard
Flag that indicates that the standard logholes (i.e. "/var/emu-
lab/logs", "/local/logs") should not be downloaded.
-P, --no-post
Do not do any additional post-processing of the log files. Cur-
rently, the only post-processing is done on the pcap files gen-
The validate action is used to check that the logs were sync'd cor-
rectly. Currently, the following checks are performed:
program-agent logs
The stdout and stderr logs from program agents are checked by
comparing their metadata against that saved in the accompanying
".status" files.
valid soft links
All soft links are checked to ensure the referent exists.
The archive action is used to archive the logs in an experiment's log
directory for future reference. The action will produce a standard zip
archive with the logs and some metadata about the experiment and when
it can be garbage collected.
Available options:
-k, --keep-until=(i-delete|space-is-needed)
Keep the archive until you decide to delete it manually or space
is needed. See the GC section later in the manual to learn how
this option and others affect garbage collection. (Default:
space-is-needed)
-a, --keep-atleast=N
Keep the archive atleast N days after creation. This value
keeps the archive from being garbage collected when more space
is needed for atleast the given number of days. (Default: 3
days)
-c, --comment=COMMENT
Add a comment to the archive. This option can be used multiple
times to add more than one comment to the archive. The comments
will be displayed by the show action and can be useful for stor-
ing information about the experiment, for example, the input
parameters. If the argument to this option is is a single dash
(-) the comment will be read from standard in.
-d, --delete
After creating the archive, mark the experiment as clean-on-sync
so that any stale data is cleaned out before performing the next
sync operation. Currently, the mark is a file named ".cleanon-
sync" in the experiment log directory.
The change action is used to change the metadata of an existing ar-
chive. For example, if after analyzing the log files, you decide that
they represent "good" data, you can add a comment stating that fact and
mark the archive as not garbage collectable. The action takes the same
set of options as the archive action.
-X Only list archives that are ready to be garbage collected.
-o List archives that do not match the above flags. In other
words, archives that will not be deleted at the next garbage
collection and are more than a day away from their keep-atleast
dates.
-m, --keep-atmost=N
Specify how many archives should be kept in the experiment.
This setting effects what files will be garbage collected, so
you should pass this same option to the gc if you use a differ-
ent value from the default of 100 archives.
-s, --keep-size=megabytes
Specify the maximum total size, in megabytes, for all of the ar-
chives in the experiment. This setting effects what files will
be garbage collected, so you pass this same option to the gc if
you use a different value from the default of 3MB.
The show action provides a more detailed listing of the log archives
for an experiment. The listing contains information about when and who
created the archive, any attributes used when computing the GC status
of the archive, comments attached to the archive, and a listing of the
files in the archive.
Optional arguments:
archive-name
The full or partial name of the archive to display. If a par-
tial name is given, any archive names that start with the argu-
ment are displayed. The default behavior is to display all of
the archives in an experiment.
The clean action is used to clean out log files from the experiment's
log directory and the log directories on the nodes. The default action
is to ask confirmation and then clean out all of the log files.
Available clean options:
-f, --force
Do not prompt for confirmation.
-n, --nodes
Only remove log files on the nodes.
-e, --experiment
Only remove log directories in the experiment's log directory.
-R, --root
Use sudo to run the clean as root on the nodes
3. The oldest files that are marked 'keep until "space-is-needed"'
will be deleted until the keep-atmost and keep-size conditions
are met or there are no more files that can be deleted without
user intervention.
Available gc options:
-m, --keep-atmost=N
Specify how many archives should be kept in the experiment.
(Default: 100 archives)
-s, --keep-size=megabytes
Specify the maximum total size, in megabytes, for all of the ar-
chives in the experiment. (Default: 3.0 MB)
By default, the project and experiment ID will be inferred from the
current working directory, if it is inside the experiment's directory
(i.e. /proj/pid/exp/eid). This behavior can be overridden using the -e
option.
3 If rsync reports an error.
2 If there was an error processing the command line arguments.
0 If the action was completed successfully.
To synchronize the log directory for experiment "neptune/myexp" with
the log holes on the experimental nodes.
[vmars@users ~] loghole -e neptune/myexp sync
To archive the newly recovered logs and print out just the name of the
new log file:
[vmars@users ~] loghole -e neptune/myexp -q archive
/proj/pid/exp/eid/logs
The log directory for an experiment.
/proj/pid/exp/eid/logs/node
The log directory for a node in the experiment.
/proj/pid/exp/eid/logs/link
The log directory for a traced LAN or link. The symbolic links
in these directories refer to the pcap(3) files for a node con-
nected to this LAN or link.
cally sync'd. This directory usually holds logs generated by
the Emulab software running on the node.
event-sched(8), tevc(1), zip(1), rsync(1), pcap(3), mergecap(1), tcp-
trace(1), xplot(1)
The Emulab project at the University of Utah.
The Emulab project can be found on the web at http://www.emulab.net
Emulab June 16, 2005 LOGHOLE(1)