Using this utility to manage an experiment's log files is encouraged because it will transfer the logs in a network-friendly manner and is already integrated with the rest of Emulab. For example, any programs executed using the Emulab event-system will have their standard output/error automatically placed in the "/local/logs" directory. The tool can also be used to preserve multiple trials of an experiment by producing and managing zip archives of the logs.
You can learn more about the loghole utility by reading its man page on users.emulab.net, included below.
loghole [-hVdqva] [-e pid/eid] [-s server] [-P port] action [...] loghole sync [-nPs] [-r remotedir] [-l localdir] [node1 node2 ...] loghole validate loghole archive [-k (i-delete|space-is-needed)] [-a days] [-c comment] [-d] [archive-name] loghole change [-k (i-delete|space-is-needed)] [-a days] [-c comment] archive-name1 [archive-name2 ...] loghole list [-O1!Xo] [-m atmost] [-s megabytes] loghole show [archive-name] loghole clean [-fne] [node1 node2 ...] loghole gc [-n] [-m atmost] [-s megabytes]
The loghole utility downloads log files from certain directories on the experimental nodes (e.g. "/local/logs") to the Emulab users machine. After downloading, it can also be used to produce and manage archives of the log files. Using this utility to manage an experiment's log files is encouraged because it will transfer the logs in a network- friendly manner and is already integrated with the rest of Emulab. For example, any programs executed using the Emulab event-system will have their standard output/error automatically placed in the "/local/logs" directory. The tool can also be used to preserve multiple trials of an experiment by producing and managing zip archives of the logs. The set of logs that are actually downloaded by the tool are those located in logholes on the nodes, where a loghole is simply a well- known directory that acts like a blackhole for log files. Any files found in these directories are downloaded to the experiment's log directory (i.e. "/proj/<pid>/exp/<eid>/logs") and placed under separate directories for each node and loghole. The referent of symbolic links are also downloaded, so if you do not want an entire directory down- loaded, you can create links in a loghole to those files of interest. To perform its various tasks, the loghole utility is broken up into several sub-actions that you can apply to an experiment's log holes or log archives. As a quick example, to synchronize the logholes for the experiment "neptune/myexp" and create an archive of these logs you would run: [vmars@users ~] loghole -e neptune/myexp sync [vmars@users ~] loghole -e neptune/myexp archive The following listing gives a brief overview of the current set of list Print a brief listing of the archives in the experiment's log directory. show Print a detailed listing of the archives in the experiment's log directory. clean Clean out the experiment log directory by removing any subdirec- tories and/or clean the log directories on the nodes. gc Garbage collect old archives to free up disk space. Options passed to the utility are split between globally applicable ones that are placed before the action and action-specific options that are placed after. For example, the -a global option will apply an action to all of your experiments and the -X option for the list action will list only those files that will be deleted at the next garbage collection. To combine these options you would enter: [vmars@users ~] loghole -a list -X The set of global options is as follows: -h, --help Print the usage message for the loghole utility as a whole or, if an action is given, the usage message for that action. -V, --version Print out version information and exit. -d, --debug Output debugging messages. -q, --quiet Decrease the level of verbosity, this is subtractive, so multi- ple uses of this option will make the utility quieter and qui- eter. The default level of verbosity is human-readable, below that is machine-readable, and below that is silent. For exam- ple, the default output from the "list" action looks like: [ ] foobar.1.zip 10/15 [!] foobar.0.zip 10/13 Using a single "-q" option changes the output to look like: foobar.1.zip foobar.0.zip -e, --experiment=PID/EID Specify the experiment(s) to operate on. If multiple -e options are given, the action will apply to all of them. This option overrides the default behavior of inferring the experiment from
The sync action is used to synchronize the logholes on the nodes with the experiment's log directory on the Emulab users machine. The action will iterate through each node in the experiment and perform an rsync(1) on the loghole directories for that node. Currently, the default set of directories that will be synced are "/var/emulab/logs" and "/local/logs". In addition, if any of the network links in the experiment are being traced, loghole will perform the following extra steps: 1. Create a directory for each link. 2. Setup symbolic links to the pcap(3) files retrieved from the delay nodes. 3. Run the pre-shaped pcap(3) files (e.g. *.recv.0) through tcp- trace(1) to generate graphs viewable with xplot(1). Note that the processing will only be done for pcap files that were gener- ated by a SNAPSHOT event sent to the tracemon agent. The "-s" option is provided to automatically send this event for each agent. While the download is in progress, loghole will display some simple statistics. The left-hand side of the display shows the number of nodes remaining to be synced, in progress, and completed. The right- hand side shows minimum, average, and maximum amount of time needed to sync a node. Finally, a "spinner" on the far right is updated when the currently active rsync log files have grown, which usually happens when more files are being synced. If rsync(1) encounters an error while running, it will automatically be rerun after a short delay. Optional arguments: -r, --remote=remotedir An additional remote directory to sync. This option is addi- tive, so you can download several additional directories. -l, --local=localdir The local directory to store the downloaded files. This defaults to the experiment's log directory. -n, --no-standard Flag that indicates that the standard logholes (i.e. "/var/emu- lab/logs", "/local/logs") should not be downloaded. -P, --no-post Do not do any additional post-processing of the log files. Cur- rently, the only post-processing is done on the pcap files gen-
The validate action is used to check that the logs were sync'd cor- rectly. Currently, the following checks are performed: program-agent logs The stdout and stderr logs from program agents are checked by comparing their metadata against that saved in the accompanying ".status" files. valid soft links All soft links are checked to ensure the referent exists.
The archive action is used to archive the logs in an experiment's log directory for future reference. The action will produce a standard zip archive with the logs and some metadata about the experiment and when it can be garbage collected. Available options: -k, --keep-until=(i-delete|space-is-needed) Keep the archive until you decide to delete it manually or space is needed. See the GC section later in the manual to learn how this option and others affect garbage collection. (Default: space-is-needed) -a, --keep-atleast=N Keep the archive atleast N days after creation. This value keeps the archive from being garbage collected when more space is needed for atleast the given number of days. (Default: 3 days) -c, --comment=COMMENT Add a comment to the archive. This option can be used multiple times to add more than one comment to the archive. The comments will be displayed by the show action and can be useful for stor- ing information about the experiment, for example, the input parameters. If the argument to this option is is a single dash (-) the comment will be read from standard in. -d, --delete After creating the archive, mark the experiment as clean-on-sync so that any stale data is cleaned out before performing the next sync operation. Currently, the mark is a file named ".cleanon- sync" in the experiment log directory.
The change action is used to change the metadata of an existing ar- chive. For example, if after analyzing the log files, you decide that they represent "good" data, you can add a comment stating that fact and mark the archive as not garbage collectable. The action takes the same set of options as the archive action. -X Only list archives that are ready to be garbage collected. -o List archives that do not match the above flags. In other words, archives that will not be deleted at the next garbage collection and are more than a day away from their keep-atleast dates. -m, --keep-atmost=N Specify how many archives should be kept in the experiment. This setting effects what files will be garbage collected, so you should pass this same option to the gc if you use a differ- ent value from the default of 100 archives. -s, --keep-size=megabytes Specify the maximum total size, in megabytes, for all of the ar- chives in the experiment. This setting effects what files will be garbage collected, so you pass this same option to the gc if you use a different value from the default of 3MB.
The show action provides a more detailed listing of the log archives for an experiment. The listing contains information about when and who created the archive, any attributes used when computing the GC status of the archive, comments attached to the archive, and a listing of the files in the archive. Optional arguments: archive-name The full or partial name of the archive to display. If a par- tial name is given, any archive names that start with the argu- ment are displayed. The default behavior is to display all of the archives in an experiment.
The clean action is used to clean out log files from the experiment's log directory and the log directories on the nodes. The default action is to ask confirmation and then clean out all of the log files. Available clean options: -f, --force Do not prompt for confirmation. -n, --nodes Only remove log files on the nodes. -e, --experiment Only remove log directories in the experiment's log directory. -R, --root Use sudo to run the clean as root on the nodes 3. The oldest files that are marked 'keep until "space-is-needed"' will be deleted until the keep-atmost and keep-size conditions are met or there are no more files that can be deleted without user intervention. Available gc options: -m, --keep-atmost=N Specify how many archives should be kept in the experiment. (Default: 100 archives) -s, --keep-size=megabytes Specify the maximum total size, in megabytes, for all of the ar- chives in the experiment. (Default: 3.0 MB)
By default, the project and experiment ID will be inferred from the current working directory, if it is inside the experiment's directory (i.e. /proj/pid/exp/eid). This behavior can be overridden using the -e option.
3 If rsync reports an error. 2 If there was an error processing the command line arguments. 0 If the action was completed successfully.
To synchronize the log directory for experiment "neptune/myexp" with the log holes on the experimental nodes. [vmars@users ~] loghole -e neptune/myexp sync To archive the newly recovered logs and print out just the name of the new log file: [vmars@users ~] loghole -e neptune/myexp -q archive
/proj/pid/exp/eid/logs The log directory for an experiment. /proj/pid/exp/eid/logs/node The log directory for a node in the experiment. /proj/pid/exp/eid/logs/link The log directory for a traced LAN or link. The symbolic links in these directories refer to the pcap(3) files for a node con- nected to this LAN or link. cally sync'd. This directory usually holds logs generated by the Emulab software running on the node.
event-sched(8), tevc(1), zip(1), rsync(1), pcap(3), mergecap(1), tcp- trace(1), xplot(1)
The Emulab project at the University of Utah.
The Emulab project can be found on the web at http://www.emulab.net Emulab June 16, 2005 LOGHOLE(1)