The d0rte package - Status and Demo: d0scan_qt, D0reco_x

David J. Ritchie
http://www-d0.fnal.gov/~ritchie/ADMdemo.html
October 18, 2002

Purpose

The purpose of the DZero run time environment product (d0rte) is to make it easy to create a run time environment directory that contains all the files necessary to run a particular D-Zero executable that is part of the D0RunII Release. The packages from which the executable and the run time files come must conform to certain specifications. Once they do, it becomes a straight-forward, almost clerical process, to create a run time environment directory that can run the executable in conjunction with whatever input data it requires.

Acknowledgements

All of the concept and much of the implementation of d0rte has been due to Jonathan Hays.  Paul Russo and Alan Jonckheere have provided extensive support, consultation, and advice. Laurent Duflot has made a run time environment using d0rte that includes the D0scan_qt event display.

Demonstration - d0scan_qt

The d0scan_qt package has been adapted to conform to the d0rte specifcations by Laurent Duflot. It takes advantage of much of the work that was done by David Ritchie to adapt D0reco_x to the d0rte. The packages that supply runtime data for D0reco_x have also been modified: sftalign, empartfit, tau_hmchisq, run_config_mgr, fps_unpdata, and smtraw. Laurent "grandfathered" the magnetic field data and geometry_db in order to make his run time environment.  (The geometry_db package has since been updated to conform to the d0rte rules.)

(Over to Laurent.) 

Laurent reports that it took him approximately 4 hours to modify the d0scan_qt package to conform to the d0rte rules. The resulting runtime environment which includes the d0scan_qt executable is about 600 Megabytes. The size in regular operation would be somewhat smaller because trigsim is presently always included whether one wants it or not. If this were removed, the size would decrease by about 200 Megabytes.

Laurent sees this as allowing one to tar up runtime environment and send it to another computer where someone could employ it to view events. We believe that we have included everything in the environment that is necessary to run the executable on another (linux or IRIX) computer that is typically used at D0.

Concept of Operation

The concept of operation is one of publish and subscribe. Packages which supply runtime data to any executable publish their data in the D0RunII directory rundata. Packages (such as d0reco or d0scan_qt) which have executables that make use of that data subscribe to it. Because executables expect to get their data in a number of different ways, an executable implements its subscription by providing shell scripts that
get the data and/or establish environmental variables that point to it.

The steps are that a person...

Demonstration - D0reco_x - Nitty-Gritty

The following shows a step by step log of doing the same thing but for D0reco_x. At the time this log was recorded, there were still missing aspects to the way the d0rte handled RCP's.  That has since been corrected.  Presently, it remains to "teach" some of the packages, such as d0gstar and mag_field, how to "publish" their data into the rundata area. The work that Laurent has done should contribute to that. We expect that it will be possible to build a run time environment with which one can run D0reco_x in a very short while.

Building a Local Release

We follow the usual steps to build a local release on d0lxbld5. Our local release is called MyCPBDemo0234xa (to identify it for record keeping purposes to be, in addition to the CPB Demo, a local release based on t02.34.00 on Linux attempt a. 
Terminal type is xterm
/usr/bin/find: *: No such file or directory
/usr/bin/find: *: No such file or directory
 
setup D0RunII t02.34.00
**** setup D0RunII t02.34.00

newrel -t t02.34.00 MyCPBDemo0234xa
read user srtrc
Creating a test release "MyCPBDemo0234xa" in the directory
    /home/ritchie
Linking tmp to /scratch/10/ritchie/MyCPBDemo0234xa/tmp
Linking bin to /scratch/10/ritchie/MyCPBDemo0234xa/bin
Linking lib to /scratch/10/ritchie/MyCPBDemo0234xa/lib

cd MyCPBDemo0234xa
d0setwa

pwd
/RunII/home/ritchie/MyCPBDemo0234xa

ls
bin  doc  GNUmakefile  include  lib  man  rcpdb  results  tmp

Creating a Runtime Environment Containing D0reco_x

We create a runtime environment containing D0reco_x using the d0rte product. Because d0rte is still under development, we setup the d0rte version v00-03-01 that is referenced in my private ups database in the directory ~/myupsdb. Our runtime environment directory in which everything is contained is called CPBDemo1a (to identify it for record keeping purposes to be, in addition to the CPB Demo, run time environment number 1 attempt a. 
setup d0rte v00-03-01 -z ~/myupsdb
d0rte t02.34.00 $SCRATCH/CPBDemo1a "" rcplink d0reco_x
D0RTE: start directory   = /home/ritchie/MyCPBDemo0234xa
D0RTE: release version   = t02.34.00
D0RTE: install directory = /scratch/10/ritchie/CPBDemo1a
D0RTE: rundata directory = <from private/public context>
D0RTE: rcp option        = rcplink
D0RTE: making new local release install directory:  /scratch/10/ritchie/CPBDemo1a
...
D0RTE: made architecture-specific ...bin/ and ...lib/ (if needed):  Linux2.4-KCC_4_0
D0RTE: making link to d0tools
D0RTE: beginning to make rundata
D0RTE: copying public rundata
D0RTE: Contents of /scratch/10/ritchie/CPBDemo1a/rundata:
drwxr-sr-x    2 ritchie  D0           4096 Sep 17 04:25 cft_obs
...
drwxr-sr-x    2 ritchie  D0           4096 Sep 20 21:28 tau_hmchisq
D0RTE: doing links to KAI runtime libraries
>>>RUNTIME ENVIRONMENT SET FOR KAI on Linux<<
D0RTE: linking to the official and personal rcpdb
D0RTE: Linking to official FileSystemDB
D0RTE: Generating environment variable bootstrap script
D0RTE: Generating d0tools bootstrap script
D0RTE: Running specified executable d0rte setup scripts
D0RTE: exe =  d0reco_x
D0RTE: script =  /scratch/10/ritchie/CPBDemo1a/rundata/setup-scripts/d0reco_x/dreco_x-setup.sh
d0reco_x_setup script -=-=-=-=-=-=-=-=
d0reco_x-setup.sh: Making copies of shared object libraries...
d0reco_x-setup.sh: Attempting to make a copy of the D0reco_x executable...
d0reco_x-setup.sh: Found public context executable; copying...
d0reco_x-setup.sh: Attempting to copy the public context framework D0reco rcp's
d0reco_x-setup.sh: Copying /D0/dist/releases/t02.34.00 framework D0reco rcp's
d0reco_x-setup.sh: Attempting to copy the private context framework D0reco rcp's
d0reco_x-setup.sh: No /scratch/10/ritchie/CPBDemo1a/d0reco/rcp/*.rcp files found
framework_rcps/d0reco_x/*.rcp

Running D0reco_x From Within the Runtime Environment

We prepare the input as required by the D0reco_x executable and run the executable using the script supplied by d0tools. Again, because d0rte is still under development and there have been some necessary modifications to d0tools, we setup the d0tools version that is referenced in my private ups database ~/myupsdb.

We begin with a new terminal session and do not set up anything related to the D0RunII release in order to demonstrate that we are really using the runtime environment that we have created and nothing else. 

Terminal type is xterm
/usr/bin/find: *: No such file or directory
/usr/bin/find: *: No such file or directory
 
cd $SCRATCH
mkdir CPBDemoRun1a
cd CPBDemoRun1a
cp ~/mydata.dat .
setup d0tools -z ~/myupsdb
runreco -userte \
? -rtepkg=d0reco_x \
? -rtepath=../CPBDemo1a \
? -filelist=mydata.dat \
? -format=data \
? -num=1
rund0exe: d0rte operation selected. 
rund0exe: this overides localrcp and localbuild flags
rund0exe :  using  d0reco_x
rund0exe :  using  ../CPBDemo1a
Initialising environment from ../CPBDemo1a/GlobalEnvBootstrap.sh
D0RTE_RTEDIR =  /scratch/10/ritchie/CPBDemo1a
D0RTE_ARCH   =  Linux2.4-KCC_4_0
RTEPKG       =  d0reco_x
RTEPATH      =  ../CPBDemo1a

Results for this job to be placed in ./D0reco_x--Linux-mydata
rund0exe : Sourcing initscript : /.../d0reco_x-runtime.sh 
with rundir parameter D0reco_x--Linux-mydata

d0reco_x-runtime.sh script -=-=-=-=-=-=-=-=-=-=
d0reco_x-runtime.sh: Subscribing to cft_obs data files
d0reco_x-runtime.sh: Subscribing to d0reco data files
...
d0reco_x-runtime.sh: Subscribing to tau_hmchisq data files
d0reco_x-runtime.sh : defining environmental variables needed by d0reco_x


Observing D0reco_x Executing

We observe D0reco_x executing as verified by running top in another window. Eventually, execution terminates due to an RCP problem, evidently. This is unfortunate but then the integrated test does so also.  See:
http://d0ntwg02.fnal.gov/buildlogs/viewLogFile.aspx?release=t02.34.00&platform=Linux2.4-KCC_4_0&build=3&package=d0reco&phase=test
Thus we conclude that D0reco_x under the runtime environment is as equally successful as d0reco's integrated test.

D0reco_x's Screen Output

In Register Maker : L3RunConfigMgr  *
In Register Maker : L3TCalUnp  *
In Register Maker : L3TCalCluster  *
Framework exception occurred:
Information: Framework constructor: RCP problem

D0reco_x's Log Output

Looking in the directory D0reco_x--Linux-mydata created by the runreco script as part of setting up the D0reco_x execution, we find the D0reco_x.log file and examine it to find: 
%ERLOG-A init: 
         An attempt to find a parameter set failed.
The requested package name was: d0reco
The requested object name was : reco_det_data
The name of the script causing the failure is:

(if this name is NONE, then we were reading a database entry)
          Framework 24-Sep-2002 14:38:54  -:- -
%ERLOG-A init: Framework constructor: RCP problem
          Framework 24-Sep-2002 14:38:54  -:- -

Conclusion

As Laurent has shown, the d0rte product is ready for serious effort on the part of beta testers. It needs to be quickly checked out and put through its paces and then placed into production for widespread use. In principal, it should be of interest to data analysis efforts on farms, Monte Carlo activities and the running of other executables.

There will no doubt be areas in need of improvement. Feedback of experiences that are the result of aggressive deployment will allow the addressing of the areas of deficiency that almost certainly exist.

It is crucial: