Readme for pav-0.1:

Idea:

This package is meant to help with the setup and shutdown of PVFS
volumes.  It was written with the intent of making the testing of such
volumes simple, since testing PVFS volumes often involves the
time consuming process of making changes to PVFS server code, pushing
the compiled binaries out to server nodes, setting up PVFS server
config files, and finally starting the iod and mgr processes.  The pav
package is also useful for users who would like to set up PVFS volumes
on systems which they only have user accounts on.  Such a user could
install the pav package in their home directory and use it inside of a
PBS script to create a useable PVFS volume that only persists for the
duration of their job.  Sysadmins like the author also use the package
to push out new versions of PVFS code onto their cluster wide PVFS
resource.

Typically, PAV is used in two environments.  In one case, a user has a
handful of workstations.  Using PAV, he can get a mulit-server PVFS2
volume up and running and quickly start testing.  In the other common
case, the user requests an allocation of nodes on a cluster.  With PAV,
the allocation is split into "io nodes" and "compute nodes".  PVFS2
servers run on the io nodes while the application runs on the compute
nodes.  This split keeps the application nodes from perturbing the io
nodes and should yield more consistent results.

PAV can also mount PVFS, but only if you have a way of obtaining root
without a passowrd.  Without speaking to the security implications, you
could achieve this with ssh keys or sudo configurations.   See the
config options below for how you would enable this optional feature.

Using pav:

To use the system (after installation), one must follow these steps:

1.) create a list of machine 'resources'.  This is simple a file
containing a list of machines, one per line.  Exactly the same as a
MPICH machines file.

2.) create a pav configfile (we will call the file pav.conf but it can
be called anything).  The configfile has key=val syntax and can
contain comments (line beginning with '#').  Any value can use the
value of a previously declared key by using a $KEY syntax (see STORAGE
below).  There is a sample configfile with the distribution called
configfile.sample.

The following keys can be defined:

NODEFILE="</path/to/machinefile>"
IONCOUNT=<number of 'io' nodes for the system>
METACOUNT=<number of 'meta' nodes for the system>
UNIQUEMETA=<1 = dedicated metadata server 
		0 = metadata server runs on data server >
PROTOCOL=<'tcp' (TCP/IP), 'gm' (Myrinet), 'ib' (infinband) >

WORKINGDIR="</path/to/pvfs_datadir>" This is where all pvfs data will
be written to on the remote 'io' nodes.  Usually use a large, fast
volume for this directory *NOTE: you should always use a unique, empty
directory for this option.  A pav_stop will remove this entire
directory tree.  The same is true for STORAGE below, we usually put
STORAGE as a subdirectory of WORKINGDIR.

PVFSPORT=<port number to use>
STORAGE="</path/to/storage_adir>" Usually set to $WORKINGDIR/data
SERVERLOG="</path/to/logdir>" Usually set to $WORKINGDIR/log

MOUNTPOINT="</path/to/pvfs_virtual_mountpoint>" 
ENCODING="{direct|le_bfield|xdr}" Adds "encoding=" mount option, usually not
                                  set.


BINDIR="</path/to/dir/containing/binaries>" Usually $WORKINGDIR/bin

#USERNAME=nobody
#GROUPNAME=nobody

RCMDPROG=<rsh|ssh>
RCPPROG=<rcp|scp>

SERVER="</path/to/pvfs2-server>" This specifies the path to the pvfs2-server
binary to be pushed out to all systems and executed, this is not the
path specifying where to run the pvfs2-server from, only where to get the pvfs2-server  
to use.  Same is true for PINGPROG.
PINGPROG="</path/to/pvfs2-ping/binary>"

GENCONFIG="</path/to/pvfs2-genconfig>" pvfs2-genconfig generates the config 
  file for the pvfs2 servers.  It will run locally and then the generated files
  will later be copied to the pvfs2 nodes.  

COPYBINS=<1 = default, copy binaries out to temp area on each node,
          0 = do not copy, and run them from the specified path above>

TROVESYNC=<1 = default. sync data after metadata operations,
           0 = do not perform implicit sync. improved performance at the risk 
	   		of data loss>

TROVEMETHOD="<desired Trove method>" This defaults to 'dbpf'.

COMPUTENODES_LAST=<1 = default.  Choose compute nodes from end of host list,
                   0 = Choose compute nodes from beginning of host list>

MOUNT_FS=<0 = default. don't bother with the VFS interface
          1 = load the kernel module, start the helper apps, and mount
	      the fs>

PVFS_KMOD="<path to kernel module>"  Be sure to specify a kernel module
                                     built for the compute nodes
PVFS_CLIENT="<path to pvfs2-client>" The helper app

RCMDPROG_ROOT="<mechanism for gaining remote root>"  Could be anything
						     really.  "ssh -l
						     root" is the one
						     we've tested most.

3.) execute pav_start -c <configfile>.  You will see the pav system
setting up a PVFS volume for you.

The following command line arguments can be passed to pav_start:

	-c -config        configuration file for PAV to use
	-r -root          path to program root (default ./)
	-m -machinefile   file with list of available nodes
	-n -ionodecount   number of nodes to use for IO
	-e -mEtanodecount number of nodes to use as metadata servers
			  (wanted to use -m but it was taken by machinefile)
	-h -help          display this message

4.) execute pav_info -c <configfile>.  This command will give you lots
of information about your newly created PVFS volume, including the
PVFS2TAB_FILE.

5.) run PVFS tests.  The volume is set up and ready to use!

6.) when finished, clean up using pav_stop -c configfile.  WARNING:
this command will kill all iod/mgr processes and clean up all data
stored in the associated data directories.

Thanks and good luck!  Any questions/comments should be sent to
pvfs2-developers@beowulf-underground.org