Alan M. Jonckheere

Getting Started with Code Management at D0

This document is organized into sections from simple uses to more complex:

Introduction

The CD/D0/CDF/SDSS Run II code management system uses a set of tools, most of which are freely available on the web with some pieces home grown at FNAL. The tools used in the system include SoftRelTools originally developed by BaBar at SLAC, CVS, GNU tools, and UPS/UPD. See Code Management Tools at D0, references for links to long writeups of the basic tools being used.

The SoftRelTools suite is mostly a set of short scripts, running under the Bourne Shell plus a set of makefiles, examples, documentation and, perhaps, most importantly a prescribed directory structure. The requirements on the users are to decide the details of the lower level parts of the directory structure and to provide relatively simple makefiles, for which examples and skeletons are provided. It uses GNU make (gmake), cpp for preprocessing and g++ for C++ compilation etc.

This document attempts to give the novice user enough information with simple instructions and examples to get started using the full system to manage their code development without a lot of detail. The detail is supplied by links to various sources of "official" documentation, most of which are listed above.


Gaining read access to the repository

Read access to the CVS repository is controlled on an individual basis by one of two methods, either kerberos ("setup d0cvs") or ssh (RSA) ("setup d0sshcvs") authentication. To gain read access, you must send either your kerberos principal name or your ssh public key to d0-release-mgr@fnal.gov. Kerberos authentication is much preferred, but is only possible on a kerberized machine. All unix machines at Fermilab are kerberized.

If, after being notified that your principal or key has been installed, you are still having problems,

Read and write access are controlled separately. Any bonified member of D0 is allowed read access. Write access requires read access plus authorization to write to a specific package. The request to add you to the access list for a particular package should be sent to d0-release-mgr@fnal.gov by one of the existing authors. Make sure that the message contains both your principal and your email address.

NOTE: you do not need read access to link against any of the currently supported releases of the D0 code (see setup and simple use below). You need this access only if you need to modify a fair amount of the released code. In the latter case, it'd be easier to use the developer tools described below. These require read access, and maybe write access to the repository.


Setting up to use a particular release

The choice of the particular release version of the software you will work (link) with is accomplished by setting a number of environment variables for the release area itself, plus any that need to be set by the particular packages you'll be using. This is all done via:

setup D0RunII <version>

<version> specifies which particular release version you will be using. Unlike in the Run I system, there is no specific limit (other than disk space to store them) on the number of versions that can be available at any given time. If <version> is omitted, the release version currently declared as "current" will be setup.

To see what release versions are available, do:

setup D0RunII does a lot of things, including setting your $PATH etc to point to the appropriate release. To undo all of that, just unsetup D0RunII.

In addition to setting up a particular release version of the D0 Software (D0RunII) it is also possible, though not recommended, to setup a particular version of a package. This might be done to pickup an older version of a data file, for example. However, such a setup will not change the archive libraries that you link against. Those are a part of the release not the package. Before you setup a package, be sure you know what you're doing. To override the particular package version present in a release you'll have to use the developer tools described below.

In the C++ world, you can't "mix and match" versions of packages like we did with LIBTEST in Run I, without having to recompile (possibly large fractions of) the code. The developer tools and the SRT directory structure are designed to make this, if not easy, then at least doable within the lifespan of a graduate student. This recompilation isn't as bad as it once was, given today's disks and machines.


Version naming conventions

The <version> that you specify to setup is merely a name. We have developed a convention for these names that is aimed more at their eventual use in production releases than at their current use in a rapid development system. The <version> consists of four fields: a prefix and three two digit numbers, an example might be p11.03.01.

We currently have only a few prefixes:

In addition to specifying a particular release version directly. It is possible to use a "version alias" to specify the version. Currently only "current", meaning the latest release that is known to work, ie) has been in place for a while and "test" meaning latest completed release are available.

Individual code "packages" also have versions. These all have a prefix of "v" and the usual three two digit numeric fields. These fields have the same meaning as for a release. The only major difference between release versions and package versions is that the numeric fields are delimited by "." (dot) in releases and by "-" (dash) in packages. The latter is because cvs does not allow "." (dot) in tags, that is version names. This also helps to keep the two seperate.


Navigating through the release area

The structure of the code release area is relatively simple, and a number of environment variables, including: and others, all of which a set by the setup command above, allow you to access it all. It is organized into two main branches:

Packages are just logical groupings of code that a single developer or a small group would work on. A package is the basic unit of development. There are as many ways to organise a package as there are developers. Some prefer very small packages, but a lot of them. Others prefer larger packages organised into subdirectories, or subpackages. Almost anything is possible.

Likewise, there are several ways to organize releases. The two main possibilies are:

For details of the rest of the directory structure, please refer to SoftRelTools User's Guide/Tutorial and the SoftRelTools Intro, or just go look. Suffice it to say that many of the directories in the $SRT_DIST/releases tree are really links to directories elsewhere, either in /releases or in /packages. These are explicitly pointed out when they matter in the text below.


Using already released code

NOTE: to use the released code you do not need any access to the CVS repository.

The simplest way to use already released code is to directly use the main $SRT... environment variables from the command line, in your own scripts or makefiles. After having done the appropriate setup described above, you can use the following directory specs to find:

etc. Look through the directory trees. <pkgname> above is the name of the package of which the include file is a part.

NOTE: Many of the "directories" are actually links to other directories. For example $SRT_DIST/releases/current is a link to a specific numbered version.

Compiling and linking from the command line can then be done using the -I and -L compiler switches to specify where the compiler and loader should search for include files and object code respectively. In addition to anything that your code needs, you'd include:
-I$SRT_PUBLIC_CONTEXT/include/
to tell the compiler where to find all includes from this release and
-L$SRT_PUBLIC_CONTEXT/lib/$SRT_SUBDIR/libxxx.a
one per object library to tell the loader where to search for unsatisfied externals.

The same lines should/could be put in any scripts or makefiles that you might use. We do not intend to provide ever provide such scripts. You are encouraged to use the more elaborate, but easier to use system discribed in the next sections.


Simple Code development situation

In the case where you need to modify one or more of the released packages, or you need to link with a different version of one or more packages than was used in building the base release, you need to go into developer mode. At this point you begin using the SoftRelTools suite of tools. You will need to be able to read the source code directly from the CVS repository. You'll need read access to the repository. See: Gaining read access the repository earlier in this document.

setup the release version you wish to base your work upon and the d0 cvs repository:
> setup D0RunII <version>
> setup d0cvs or setup d0sshcvs

At this point you can view the man pages via:
> man <command>
for each of the SRT commands mentioned.

Create a working directory, say "~/workdir" and change your working directory into it:
> cd
> mkdir workdir
> cd workdir
You must now create an empty version of a release structure. This is known as a local or private release:
> newrel -t <base-release> <new-release>
ie) > newrel -t t02.11.00 mytest
This will produce a directory <new-release> = mytest in your working directory and an empty release tree below that. There will be no actual packages included, even via links. Those are produced in the next steps.

The string you specify as <base-release> is extremely important. This string is saved in a file called .base_release and is used during all gmake builds to find the release you are building against. This is independent of which version you actually have setup at the time you issue the gmake command. It is used as a "sanity" check. In C++ you generally can't "mix and match" different builds and parts of builds. In addition, gmake does not allow one to go backward in time easily. You need to be careful about this "stickyness" of the tools.

Change directory into your newly created <new-release> directory:
> cd mytest

The first time you go through this, it's a good idea to check what files have been created at each stage. So:
> ls -RFa
to see what's been created. Note, in particular, the makefile, GNUmakefile that's been copied into your top level directory. This file will steer all subsequent test builds of your work. Also, note the two files .base_release which stores the <base-release> information and .experiment which tells the procedures that you're working on D0 code. You'll normally not need to modify any of these files. However, you need to be aware that the contents of these files do effect the actions of gmake during the builds. The string in .experiment causes some D0 specific flags etc to be set. The version listed in .base_release is the release version used in all links. It overrides the version of D0RunII you have setup. This is needed because switching versions of the base release can have unpredictable consequences. If you must switch versions of the base release, you should edit this file, putting in the new version number, then do gmake clean to give you a clean slate so you can start over.

At this point:
> d0setwa
This initiallizes an empty rcp database and tells the system where the top of your private build area is located. Whenever you change to a different build area, you must cd to the top area and repeat the d0setwa. Obviously, if you login a new window, you'll need to do this in that window as well.

Now you use addpkg to fetch the code you'll be working on from the CVS repository.
> addpkg [-h] <pkgname> [<version_string>]
for example:
> addpkg -h calorimeter
> addpkg -h xxx
the -h will fetch the latest or head version of the code and is highly recommended for most use. Omitting the -h will fetch the version that the base release was build on. The result is "sticky". CVS "remembers" that you checked out a particular version. If you make changes you'll have to put them back onto a cvs branch, a topic beyond the scope of this document. See: CVS Branch Usage and Patching a production version of D0 code for a general and a more specific discussion of this sort of usage.

If you're developing new code, you'll want to use the -h. But if you're trying to fix bugs or just need to link against a package version different from the one in the base release, either omitting the -h or specifying an explicit version is what you'd want. Check the files that have been created to make sure that you've gotten what you want. In particular, notice the file include/calorimeter. This is a symbolic (soft) link to the include directory of the calorimeter package. There will be one such link for each package that you've added.

Now, assuming that no other packages depend on yours, that is, no other packages that you'll link into an executable #include any of the header files in your package, and, if your code is fortran, you haven't changed any calling sequences or include files that anyone else uses, you can:
> gmake
> gmake test
to do some directory manipulations and then build your package object libraries and/or executables. The gmake test runs the required package tests.

Now you can add new routines, edit old ones etc to your hearts content. Run gmake at any time to update the object libraries and executables etc.

There are several gmake "targets" that you need to know about because you might want to use them:

We will discuss in the section on CVS fundamentals what to do with the code once you are happy with the results of your tests.


A slightly more complex development situation

The previous example explained how to do code development in a simple situation, one where no other packages depend on the one you are working on. In the more general case, you'll be working on one or more packages, but other packages, that you subsequently use in a link, depend on yours. That is, they # include header files or Fortran include files or otherwise use code that you are modifying. In general, especially in C++, it is not safe to assume that you can use that code without recompiling it, even though you will make no changes to the dependent packages, some, or all of the code will need to be recompiled.

The question immediately arises, "How do I know that another package depends on mine?" That's easily answered using the SoftRelTools tool depend. Do man depend for details. In this case just:
>depend <mypkg> <version>
will list all packages in <version> that depend on headers in <mypkg>

In this example we will discuss briefly how to solve this problem.

Follow the previous example: Simple code development situation through where you add the packages that you'll be modifying. At this point, you'll have actual copies of all the code you'll be modifying. You could go on and addpkg all of the dependent packages. However, this uses a lot of disk space and is unnecessary. Instead, you should now add links to the particular version of the dependent packages. These need to be added in two places:

The easiest way to create those links is to use lnkpkg:
> lnkpkg [-f] pkgname [<version_number>]
for example:
> lnkpkg cppreco v00-00-01
The -f switch forces replacement of existing links. You must use this to change an existing link. In this case you'll almost certainly neet to do a gmake clean as well. If you omit the version number, lnkpkg will use the version used in the base_release specified with your newrel.

gmake will now build the object libraries and any executables for the packages you've addpkg'd as before, but will also recompile and build the object libraries and any executables for the packages you've used lnkpkg on.

The rest is the same as in the Simple development example. The only difference now is that the first time you run gmake you will see your lnkpkg'd archives being recompiled and built. After the first time, any changes in your addpkg'd package may or may not provoke recompiling routines in the lnkpkg'd packages. You will also notice that mytest/lib/$SRT_SUBDIR and the corresponding /bin directories contain the object libraries and executables created from code in your lnkpkg packages.

NOTE: this is a case where it would be very advisable to explicitly invoke the separate gmake targets:
> gmake lib
> gmake calorimeter.bin
The first gmake recreates all the required object libraries. The second links only the executable for calorimeter instead of for both calorimeter and cppreco, saving you some time since you don't need the executables for that cppreco.


CVS fundamentals

To use the D0 CVS repository, you first need read access to it. Then you
> setup d0cvs
This sets the environment variable $CVSROOT and also creates an alias for a few very handy utilities. The former specifies the location of the CVS repository. The utilities are:

Some more helpful hints: setup d0cvs set's up CVS, so you don't need to do that too.
> cvs -H gives you general usage info on switches
> cvs --help-command gives you a list of commands
>cvs <command> -H gives you usage info on a command

Once you've "checked out" a module, for example by running addpkg You just use any editor to make your mods. To make your changes permanent, if you don't have write access to the CVS module=package you'll have to send the code to the library czar. If you have write access you can use the following commands:

all of these, except rtag are from the working directory. The rtag command can be issued from anywhere.
NOTE: do not use cvs tag to tag versions. The result is not visable via cvs history.

When you are completely done: > cvs release -d module_name from the directory *above* module_name to release your checkout and to delete the working directory tree.

NOTE: cvs is "sticky". If you checkout a specific version, it'll put changes back into a "branch" off that version, not the "head" or primary branch. It'll also remember that you used this particular CVSROOT. You'll occasionally see some funny results due to this, but only if you use more than one repository as I do.

For much more info on cvs than you can possibly want, see:
http://www.loria.fr/~molli/cvs-index.html
and links therein. In particular:
http://bing.rz.uni-ulm.de/cvs/train/cvstrain.html


Creating a new CVS module (a new "package")

To create a new package a CVS module there are several steps that need to be done:


Organization and conventions of packages

There are a few organizational and programming conventions that are assumed by SRT. These include: