Testbed Master Control Daemon/Client Reference

Contents

• Introduction
• TMCC client program
• Node Setup Script
• Command Reference
  • reboot
  • status
  • ifconfig
  • accounts
  • mounts
  • delay
  • hostnames
  • rpms
  • startupcmd
  • startstatus
  • ready
  • readycount
  • log

• Introduction

  The Testbed Master Control Daemon (TMCD) is a program that runs on boss.emulab.net, and provides configuration information to Testbed nodes when they boot up. The Testbed Master Control Client (TMCC), is a small program that is installed on each node, and is used to connect to the TMCD to issue requests and get the response. In addition, Testbed nodes use the TMCC/TMCD to communicate events of interest back to the Emulab Database and to the user via the Web interface. The TMCD interface is text based; clients issue requests in the form of strings consisting of a command and an optional argument. The response is also a string, in a very generic format that can be easly parsed by any C/C++ program or shell interpreter. For example, to determine how to configure the experimental interfaces on each testbed node in your experiment when it boots, you would do the following:

  tmcc ifconfig					

  The response to this request would be:

  INTERFACE=1 INET=10.0.0.1 MASK=255.255.255.0
  INTERFACE=2 INET=10.0.1.1 MASK=255.255.255.0	

  which indicates that interfaces eth1 and eth2 (or perhaps fxp1 and fxp2) should be configured to the given IP addresses and netmasks.

• TMCC

  The TMCC is a simple client program that runs on the testbed nodes and handles the details of connecting to the TMCD, issuing the request, getting the response, and printing it out. It has been compiled on FreeBSD 4.x, Redhat Linux 6.2 and 7.1, and Netbsd 1.4, and should compile on just about any operating system. Alternatively, you can integrate the TMCC into your own programs. Briefly, the TMCC connects to port 7777 (UDP or TCP) on boss.emulab.net, writes a single string to the connection, and then waits for an optional response, which is a newline separated list of strings. The TMMC exits when the other side of the connection is closed by the TMCD. The source code for the TMCC is available upon request by sending email to Testbed Operations (testbed-ops@flux.utah.edu)

• Node Setup Script

  The Emulab versions of FreeBSD 4.5, Redhat Linux 7.1, and Netbsd 1.4 all run a setup script at bootup that uses the TMCC client to configure the node. All of the interfaces are configured, user accounts for each of the members of the project are created, NFS mounts are made, etc. These setup scripts are located in /usr/local/etc/emulab.

• Command Reference

  • reboot

    Report that a node has rebooted to the TMCD. This is an informational message that is used by the TMCD to determine when a node reboots for the first time after its disk has been reloaded. No response is returned.

  • status

    Request status information about the project and experiment that the node is currently part of. Returns the project ID, experiment ID, and the node name from the NS file that described the topology. This command is typically used to determine if the setup script needs to do any further configuration; if the node is free, then no other information is going to be provided by the TMCD. The format of the reply is one of:

    FREE
    ALLOCATED=pid/eid NICKNAME=name			

    The first form indicates that the node is not currently allocated to an experiment. The second form says that the node is running as part of the "eid" experiment in the "pid" project, and was named "name" in the NS file that described the topology.

  • ifconfig

    Request the configuration information for each of the network interfaces on the node, as determined by the topology described in the NS file, and the assignment of IP addresses to interfaces that is performed when the experiment is configured. The information that is returned is typically converted into corresponding ifconfig commands on the node. However, the information can be used in any manner that is appropriate for the operating system that is running on the node. The reply to this request is one or more lines in the following format (in the unlikely case that the topology describes a node with no network links, the response to this request will be null):

    INTERFACE=Z INET=X.X.X.X MASK=Y.Y.Y.Y MAC=AA:BB:CC:DD:EE:FF 

    Which says that the network interface with MAC address "AA:BB:CC:DD:EE:FF" is assigned to IP address "X.X.X.X" with netmask "Y.Y.Y.Y". The INTERFACE specification is currently invalid, since there no way to achieve a consistent ordering of interfaces between various operating systems. Rather, the MAC address is used to determine which interface to configure. A utility program called /etc/testbed/findif is provided to map the MAC address to an interface name suitable for use with the ifconfig program. On Redhat 7.1, the setup script would take this information and issue the following shell commands.

    iface=`/etc/testbed/findif AA:BB:CC:DD:EE:FF`
    /sbin/ifconfig $iface inet X.X.X.X netmask Y.Y.Y.Y	

  • accounts

    Request group and login account information for each of the project members of the project that the experiment is running. This information can be used to generate login accounts for project members on each of the nodes in an experiment. The Emulab versions of FreeBSD, Linux, and Netbsd all have stub password/group files that do not contain any user accounts or groups. When a node first boots after being allocated to an experiment, this command is used to find out what accounts to build. The reply to this request is one or more lines of group information, followed by one or more lines of login account information:

    ADDGROUP NAME=pid GID=XXXX
    ADDUSER  LOGIN=joe PSWD=ABCD UID=YYYY GID=XXXX ROOT=N NAME="Joe User" \
                 HOMEDIR=/users/joe GLIST=ZZZ0,ZZZ1
    

    The ADDGROUP reply gives the name of the group and the numeric gid for that group. The ADDUSER reply has the following fields:

    LOGIN		The user/account name.
    PSWD		The encrypted password string, suitable for direct insertion into the password file.
    UID		The numeric uid.
    GID		The primary group for the user, as a numeric gid.
    ROOT		Indicates whether the user should be granted root access by placing the user into the root group (wheel group on FreeBSD/NetBSD).
    NAME		The full name of the user, suitable for insertion into the gecos field of the user's password entry.
    HOMEDIR		The absolute path to be used for the home directory.
    GLIST		A (possibly null) comma separated list of auxiliary group ids, as numeric gids.

    On Linux, this information would be converted into the following commands:

    groupadd -g XXXX pid
    useradd -u YYYY -g XXXX -p ABCD -G root,ZZZ0,ZZZ1 -d /users/joe -c "Joe User" joe
    

  • mounts

    Request the list of remote directories that need to be NFS mounted on the node when it boots. The reply to this request is one or more lines in the following format:

    REMOTE=fs.emulab.net:/users/joe LOCAL=/users/joe
    REMOTE=fs.emulab.net:/proj/testbed LOCAL=/proj/myproj		

    On Linux, this information would be converted into the following commands:

    mkdir /users/joe
    /sbin/mount fs.emulab.net:/users/joe /users/joe
    mkdir /proj/myproj
    /sbin/mount fs.emulab.net:/proj/myproj /proj/myproj		

  • rpms

    Request the list of RPMs that should be installed on the node when it boots, as specified in the NS file on a per-node basis. The reply to this request is null if there are no RPMs to install, or one or more lines in the following format:

    RPM=/path/to/name.rpm					

    On Linux and Freebsd, each RPM is installed with the rpm command, which will install the RPM only if it has not already been installed:

    rpm -i /path/to/name.rpm				

  • startupcmd

    Request the name of the startup script (or program) that should be run when the node boots, as specified in the NS file on a per-node basis. The reply to this request is null if a startup script was not specified, or a single line in the following format:

    CMD=/path/to/runme UID=joe				

    Which says to run /path/to/runme as user joe when the node boots. The UID is always the experiment creator. On FreeBSD, Linux, and NetBSD, the command is run once the node is running multiuser. If the node reboots before the experiment is terminated, the command will be run again.

  • startstatus

    Report the numeric exit value of the startupcmd back to the TMCD so that it can be recorded and displayed in the "Experiment Information" Web page. In fact, this does not have to be the result of the startupcmd, but can be the result of any application program. The intent is to report back information that can be used by the experimentor to determine when the experiment has finished. Each node reports back status individually. The format of this command is:

    tmcc startstatus XX				

    which sends the numeric value XX back to the TMCD. There is no response from the TMCD to this command.

  • ready

    Report an application level ready status back to the TMCD so that it can record it. A count of nodes (in your experiment) reporting ready is maintained by the TMCD, and is made available to nodes via the readycount request below. There is no response from the TMCD to this command.

  • readycount

    Request the count of nodes that have reported in ready with the ready command above. This is an application level count; nodes can use this as a very primitive form of synchronization to determine when all of the nodes in the experiment have started the application (say, via the startupcmd above) and have reached a point where it is necessary to wait until all of the nodes have reached the same point. The reply to this request is a single line in the following format:

    READY=N TOTAL=M				

    which says that N nodes have reported in, of a total number of M nodes in the experiment. The application can continue to poll until N==M, but be sure to add some delay between each poll to avoid livelock at the TMCD. Note that the ready count is essentially a use-once feature; The ready count cannot be reinitialized to zero since there is no actual synchronization happening. If in the future it appears that a generalized barrier synchronization would be more useful, we will investigate the implementation of such a feature.

  • hostnames

    Request information about the IP addresses and node names of all of the nodes in the experiment. The intent it to provide the ability to easily generate a suitable /etc/hosts file that allows experiments to operate using the symbolic names of the nodes (as defined in the NS file), instead of IP addresses, which are generally assigned by the configuration software, not the experimentor. Since nodes can use multiple experimental interfaces, the reply gives the IP address for each interface on each node. Additional aliases can be returned for nodes that are directly connected to the node making the hostnames request, or that the requesting node has a route to. Secondary interfaces, and interfaces that are not directly connected are named with a suffix indicating the link or LAN that interface belongs to. For historical reasons, we provide an additional alias in the form 'node-X' where X is the ordinal number of the interface. The reply to this request is one or more lines in the following format:

    NAME=nodeA-linkX IP=X.X.Y.A ALIASES='nodeA nodeA-0'
    NAME=nodeB-linkY IP=X.X.Y.B ALIASES='nodeB nodeB-0'
    NAME=nodeC-lanZ IP=X.X.Z.C ALIASES='nodeC-0'
    

    ALIASES is a space-separated list of aliases. The /etc/hosts file that would be created for this response is:

    X.X.X.A        nodeA-linkX nodeA nodeA-0
    X.X.X.B        nodeB-linkY nodeB nodeB-0
    X.X.Z.C        nodeC-lanZ nodeC-0
    

    Say that nodeA is making this request. NodeA is obviously connected to itself, so it gets an alias pointing to its own interface. NodeA is directly connected to NodeB on NodeB's linkY interface, so it too gets an alias so that an application running on nodeA can just use the name NodeB. NodeC is not directly connected to NodeA, and nodeA does not have a route to it (perhaps the network toplogy is disjoint, or perhaps routing was not enabled in the NS file,) so it does not get an alias. Note that, in the case of nodes that are not directly connected, no guarantee is made that the alias is picked for the 'nearest' interface.

  • log

    The log command can be used by an application to write a message to a log file on users.emulab.net. This is especially useful on the Sharks, most of which do not have console serial lines attached. The argument to the log command is a single string, in double quotes if operating within the shell:

    tmcc log "This is a log message"			

    The log file is stored in /proj/pid/logs/eid.log, where pid is the name of the project and eid is the name of the experiment. The file is appended to each time; it is the responsibility of the experimentor to zero the log file when done, or if a new experiment with the same name is started.