Demo of D0reco_x under d0rte for CPB

David J. Ritchie
September 25, 2002

Introduction

The purpose of the DZero run time environment product (d0rte) is to make it easy to create a run time environment directory that contains all the files necessary to run a particular D-Zero executable that is part of the D0RunII Release. The packages from which the executable and the run time files come must conform to certain specifications. Once they do, it becomes a straight-forward, almost clerical process, to create a run time environment directory that can run the executable in conjunction with whatever input data it requires.

Acknowledgements: All of the concept and much of the implementation of d0rte has been due to Jonathan Hays. Paul Russo and Alan Jonckheere have provided extensive support, consultation, and advice.

As a demonstration and existence proof, the d0reco package that is a part of t02.29.00 and following D0RunII releases has been modified to conform to the d0rte specifcations. The following packages in t02.29.00 and subsequent releases of D0RunII which provide runtime data for d0reco have also been modified: sftalign, empartfit, tau_hmchisq, run_config_mgr, fps_unpdata.

The concept of operation is one of publish and subscribe. Packages which supply runtime data to any executable publish their data in the D0RunII directory rundata. Packages (such as d0reco) which have executables that make use of that data subscribe to it. This is done in a maximally flexible way by the various packages providing specified scripts to copy the data into rundata and so forth. The release process and the d0rte product both invoke these scripts to obtain the runtime data and the executables which are desired to be part of the run time environment that is constructed.

The steps of the demo are:

building a local release
creating a run time environment containing the d0reco executable and needed run time files
running d0reco using the created runtime environment
observing d0reco executing

Before starting the demo, we look at the error page for the t releases on the Release Status Page:

http://www-d0.fnal.gov/computing/releases/ReleaseStatus/html/errors.html

to find a release which has successfully built a D0reco_x executable. By following the link

http://d0ntwg02.fnal.gov/buildlogs/viewBuild.aspx?release=t02.34.00&platform=Linux2.4-KCC_4_0

to the error log web page provided by Gordon Watts we see that t02.34.00 has built the D0reco_x executable. Thus, we proceed to build a local release of t02.34.00. Note, however, that the integrated tests fail with an RCP problem. To look ahead, we will find that our executable running in its own run time environment only, fails in the same way. No surprise!

For the purposes of demoing d0rte, we consider this a success.

Building a Local Release

We follow the usual steps to build a local release on d0lxbld5. Our local release is called MyCPBDemo0234xa (to identify it for record keeping purposes to be, in addition to the CPB Demo, a local release based on t02.34.00 on Linux attempt a.

Terminal type is xterm
/usr/bin/find: *: No such file or directory
/usr/bin/find: *: No such file or directory
 
setup D0RunII t02.34.00
**** setup D0RunII t02.34.00

newrel -t t02.34.00 MyCPBDemo0234xa
read user srtrc
Creating a test release "MyCPBDemo0234xa" in the directory
    /home/ritchie
Linking tmp to /scratch/10/ritchie/MyCPBDemo0234xa/tmp
Linking bin to /scratch/10/ritchie/MyCPBDemo0234xa/bin
Linking lib to /scratch/10/ritchie/MyCPBDemo0234xa/lib

cd MyCPBDemo0234xa
d0setwa

pwd
/RunII/home/ritchie/MyCPBDemo0234xa

ls
bin  doc  GNUmakefile  include  lib  man  rcpdb  results  tmp

Creating a Runtime Environment Containing D0reco_x

We create a runtime environment containing D0reco_x using the d0rte product. Because d0rte is still under development, we setup the d0rte version v00-03-01 that is referenced in my private ups database in the directory ~/myupsdb. Our runtime environment directory in which everything is contained is called CPBDemo1a (to identify it for record keeping purposes to be, in addition to the CPB Demo, run time environment number 1 attempt a.

setup d0rte v00-03-01 -z ~/myupsdb
d0rte t02.34.00 $SCRATCH/CPBDemo1a "" rcplink d0reco_x
D0RTE: start directory   = /home/ritchie/MyCPBDemo0234xa
D0RTE: release version   = t02.34.00
D0RTE: install directory = /scratch/10/ritchie/CPBDemo1a
D0RTE: rundata directory = <from private/public context>
D0RTE: rcp option        = rcplink
D0RTE: making new local release install directory:  /scratch/10/ritchie/CPBDemo1a
...
D0RTE: made architecture-specific ...bin/ and ...lib/ (if needed):  Linux2.4-KCC_4_0
D0RTE: making link to d0tools
D0RTE: beginning to make rundata
D0RTE: copying public rundata
D0RTE: Contents of /scratch/10/ritchie/CPBDemo1a/rundata:
drwxr-sr-x    2 ritchie  D0           4096 Sep 17 04:25 cft_obs
...
drwxr-sr-x    2 ritchie  D0           4096 Sep 20 21:28 tau_hmchisq
D0RTE: doing links to KAI runtime libraries
>>>RUNTIME ENVIRONMENT SET FOR KAI on Linux<<
D0RTE: linking to the official and personal rcpdb
D0RTE: Linking to official FileSystemDB
D0RTE: Generating environment variable bootstrap script
D0RTE: Generating d0tools bootstrap script
D0RTE: Running specified executable d0rte setup scripts
D0RTE: exe =  d0reco_x
D0RTE: script =  /scratch/10/ritchie/CPBDemo1a/rundata/setup-scripts/d0reco_x/dreco_x-setup.sh
d0reco_x_setup script -=-=-=-=-=-=-=-=
d0reco_x-setup.sh: Making copies of shared object libraries...
d0reco_x-setup.sh: Attempting to make a copy of the D0reco_x executable...
d0reco_x-setup.sh: Found public context executable; copying...
d0reco_x-setup.sh: Attempting to copy the public context framework D0reco rcp's
d0reco_x-setup.sh: Copying /D0/dist/releases/t02.34.00 framework D0reco rcp's
d0reco_x-setup.sh: Attempting to copy the private context framework D0reco rcp's
d0reco_x-setup.sh: No /scratch/10/ritchie/CPBDemo1a/d0reco/rcp/*.rcp files found
framework_rcps/d0reco_x/*.rcp

Running D0reco_x From Within the Runtime Environment

We prepare the input as required by the D0reco_x executable and run the executable using the script supplied by d0tools. Again, because d0rte is still under development and there have been some necessary modifications to d0tools, we setup the d0tools version that is referenced in my private ups database ~/myupsdb.

We begin with a new terminal session and do not set up anything related to the D0RunII release in order to demonstrate that we are really using the runtime environment that we have created and nothing else.

Terminal type is xterm
/usr/bin/find: *: No such file or directory
/usr/bin/find: *: No such file or directory
 
cd $SCRATCH
mkdir CPBDemoRun1a
cd CPBDemoRun1a
cp ~/mydata.dat .
setup d0tools -z ~/myupsdb
runreco -userte \
? -rtepkg=d0reco_x \
? -rtepath=../CPBDemo1a \
? -filelist=mydata.dat \
? -format=data \
? -num=1
rund0exe: d0rte operation selected. 
rund0exe: this overides localrcp and localbuild flags
rund0exe :  using  d0reco_x
rund0exe :  using  ../CPBDemo1a
Initialising environment from ../CPBDemo1a/GlobalEnvBootstrap.sh
D0RTE_RTEDIR =  /scratch/10/ritchie/CPBDemo1a
D0RTE_ARCH   =  Linux2.4-KCC_4_0
RTEPKG       =  d0reco_x
RTEPATH      =  ../CPBDemo1a

Results for this job to be placed in ./D0reco_x--Linux-mydata
rund0exe : Sourcing initscript : /.../d0reco_x-runtime.sh 
with rundir parameter D0reco_x--Linux-mydata

d0reco_x-runtime.sh script -=-=-=-=-=-=-=-=-=-=
d0reco_x-runtime.sh: Subscribing to cft_obs data files
d0reco_x-runtime.sh: Subscribing to d0reco data files
...
d0reco_x-runtime.sh: Subscribing to tau_hmchisq data files
d0reco_x-runtime.sh : defining environmental variables needed by d0reco_x

Observing D0reco_x Executing

We observe D0reco_x executing as verified by running top in another window. Eventually, execution terminates due to an RCP problem, evidently. This is unfortunate but then the integrated test does so also. See:

http://d0ntwg02.fnal.gov/buildlogs/viewLogFile.aspx?release=t02.34.00&platform=Linux2.4-KCC_4_0&build=3&package=d0reco&phase=test

Thus we conclude that D0reco_x under the runtime environment is as equally successful as d0reco's integrated test.

D0reco_x's Screen Output

In Register Maker : L3RunConfigMgr  *
In Register Maker : L3TCalUnp  *
In Register Maker : L3TCalCluster  *
Framework exception occurred:
Information: Framework constructor: RCP problem

D0reco_x's Log Output

Looking in the directory D0reco_x--Linux-mydata created by the runreco script as part of setting up the D0reco_x execution, we find the D0reco_x.log file and examine it to find:

%ERLOG-A init: 
         An attempt to find a parameter set failed.
The requested package name was: d0reco
The requested object name was : reco_det_data
The name of the script causing the failure is:

(if this name is NONE, then we were reading a database entry)
          Framework 24-Sep-2002 14:38:54  -:- -
%ERLOG-A init: Framework constructor: RCP problem
          Framework 24-Sep-2002 14:38:54  -:- -

Conclusion

The d0rte product is ready for serious effort on the part of beta testers. It needs to be quickly checked out and put through its paces by those individuals and then placed into production for widespread use. In principal, it should be of interest to data analysis efforts on farms, Monte Carlo activities and the running of other executables.

There will no doubt be areas in need of improvement. Aggressive deployment is the key to addressing the areas of deficiency that almost certainly exist.

It is crucial:

that all those providing runtime data for any D0RunII executable do so in conformance with d0rte publish rules,
that all those developing any executable that is within the D0RunII release take the necessary steps to allow the executable to be run following d0rte procedures, and, in general,
that the D0 collaboration embrace, adopt and adapt d0rte to its current and future D0RunII activiites.

Without this, the current capability demonstrated above will quickly diminish and decay.