Using SAM within D0 Framework

1. Introduction

A straightforward interface to the SAM system is provided through the D0 Framework. The package which allows any Framework application to interact with SAM is called SAMManager. Its main purpose is to provide the application with the location of the next input file by communicating with the SAM Project Master. It also determines the location of the output files created by the application (if the output buffer space is managed by SAM), and generates meta-data for those files. If the output file is to be stored and catalogued with SAM in real time, SAMManager will initiate file store after file closing.

2. Getting started: running an interactive demo

The easiest way to get familiar with SAMManager is to see an example of a working application. This can be accomplished by logging into the D0 Central Analysis Server

After running the above commands one might expect to see the output similar to the one given here. Note that in the same directory one can find several other demo scripts which use the sam submit command: submit2sam, submit2sam_script and submit2sam_parallel_script. All of those submit the SAM demo job to the batch system: the first one submits it as a Framework executable, while the other two submit it is as user provided scripts runsam_batch (for running a single user process), and runsam_parallel_batch (for running a multiple user processes in parallel).

3. Understanding the example

The only SAM command used in submit2sam scripts (sam submit) is described in the document on running SAM projects. In the following we describe the framework-related SAM issues.

The test executable SAMTest uses SAMManager to retrieve an input file, it reads and dumps all events using ReadEvent and DumpEvent packages,  and writes them into an output file using WriteEvent package. Upon its closing, SAMManager catalogues the newly created output file in the SAM database, and initializes its storing.

All of the packages used in SAMTest are listed in the top level RCP file SAMTest.rcp.

Behavior of all individual packages is controlled by their respective RCP files: ReadEvent.rcp,  WriteEvent.rcp, DumpEvent.rcp (all located in io_packages/rcp subdirectory of demo), and SAMManager.rcp (in sam_manager/rcp).

In order to use ReadEvent with SAMManager, its RCP parameter InputFile has to be set to SAMInput:, a pseudo-file which indicates that SAM will be used to dynamically generate list of input files.

// ----------------------------------------------------------------

Similarly, WriteEvent will work with SAMManager only if its RCP parameter OutputFile begins with SAMOutput: or SAMGenerated:.  In the case of SAMOutput:  SAMManager will replace this string with the name of the directory to write output file into (working directory at the moment), while the remaining file name expanders, such as the standard D0 output file name generator, are responsible for the rest. On the other hand, if SAMGenerated: is used SAMManager will generate the name of the output file.
 

The RCP file for the DumpEvent package determines the frequency and the total number of events to dump.
 

The only parameter in the SAMManager RCP file which always has to be present is PackageName. If sam submit command is used, AnalysisProject and Station will be passed into the users' executable as environment variables. The user also has to provide ApplicationName, ApplicationVersion and (optionally) WorkingGroup. Valid ApplicationName/ApplicationVersion pairs can be found on the SAM Data Browsing page. The remaining parameters are all optional: ProcessDescription  is the description of the user application, FileParentageMode controls whether output file parentage is determined by opened (default) or closed input files, while FileStoreLocation and DataTier, will be used for storing output files. If FileStoreLocation is not used, or if invalid location is provided (use the SAM Data Browsing page for the list of valid locations), the storing of output files will be disabled, but metadata for the output file will be generated and printed to to the standard output, as well as stored into a python file that can be submitted to the SAM File Storage server using the sam store command. SAMManager is using the asynchronous mode of storing output files. It will submit a file store request to the File Storage Server, and will not wait for callbacks. FileRequestTimeout is timeout (in minutes) for getting an input file. The Verbose flag is used to control the amount of messages sent to the standard output. If for any reason SAMManager has to be disabled, an optional UseSAM flag can be set to 0.   
 

Some of the parameters controlling SAMManager can also be supplied on the command line, in which case they override parameters given in the RCP file. For example, if one runs the test executable using

SAMTest  -project $SAM_PROJECT -station $SAM_STATION -rcp SAMTest.rcp

then the RCP parameters AnalysisProject and Station would be ignored. If some of the required parameters are not supplied in the SAMManager RCP file or as command line options, SAMManager will attempt to determine them from corresponding environent variables. All valid RCP parameters that have corresponding command line options and environment variables are listed below.

4. Compiling SAMManager package

Compilation of SAMManager package and the test executable should be straightforward.  On d0mino the necessary steps are the following:
 

Upon compilation, SAMManager library and test executable should be located in lib/$BFARCH and bin/$BFARCH subdirectories of testdir.
 
 

5. Compiling applications

In order to use SAMManager users' applications should be linked with the SAMManager library (sam_manager), as well as with several other SAM (sam_corba and sam_util) and CORBA libraries (corba_util, OB, JTC and CosNaming). Also, the file RegSAMManager.o should be included in the list of object files. An example below is the GNUmakefile used for compilation of the SAMManager test executable:

// ----------------------------------------------------------------
// GNUmakefile for SAMTest
// ----------------------------------------------------------------

# override CXX = purify KCC

override LOADLIBES += \
  -lsam_manager \
  -lsam_client_cpplib \
  -lsam_mis_cpplib \
  -lsam_corba \
  -lsam_util \
  -lsam_station \
  -lsam_db_server \
  -lsam_common \
  -lsam_mis \
  -lcorba_util \
  -lcorba_common \
  -lOB \
  -lJTC \
  -lCosNaming \
  -lio_packages \
  -lstream_ds \
  -ld0om_ds \
  -lstream \
  -levpack \
  -ldspack \
  -lframework \
  -lfwkprofiling \
  -lrcp \
  -ledm \
  -lidentifiers \
  -ld0om \
  -lNameTrans \
  -ld0_util \
  -lZMtools \
  -lExceptions \
  -lErrorLogger \
  -lErrLogEx \
  -lZMutility \
  -lZMTimer \
  -lCLHEP \
  -lrelversion \
  -lconfig_base \
  -lrun_config_fwk \
  -lmemutil \
  -ldl

include SoftRelTools/arch_spec_f77.mk
include SoftRelTools/arch_spec_zlib.mk

override LDFLAGS += -L$(ORBACUS_LIBDIR) -L$(SAM_LIB_LIBDIR) \
  -L$(CORBA_COMMON_LIBDIR) -L$(SAM_IDL_CPPLIB_LIBDIR) \
  -L$(SAM_MIS_CPPLIB_LIBDIR) -L$(SAM_CLIENT_CPPLIB_LIBDIR)

COMPLEXBIN := SAMTest
COMPLEXBINCPPFILES :=

BINSTANDALONEOFILES := \
  framework.o \
  RegSAMManager.o \
  ReadEvent.o \
  WriteEvent.o \
  DumpEvent.o

TBINS := $(COMPLEXBIN)

.PHONY: test tbin

test:
        +@$(MAKE) -j 1 tbin
 

// ----------------------------------------------------------------
 
 

6. Possible problems

All SAMManager error messages that users may encounter within their applications are meant to be sufficiently detailed to enable one to take the appropriate action.
 
 

Project: SAM
Package: sam_manager
$Id: framework_sam_projects.html,v 1.44 2004/01/27 22:40:28 veseli Exp $

This work is part of a development project, called SAM, which consists of a number of coordinated packages each named sam_xxxx .

Notice of authorship, copyright status, and terms and conditions, should the software eventually become available for use outside Fermilab, can be found in the README and LICENCE files in the top level directory of the main sam package.

Wyatt Merritt
Security, Privacy, Legal
Last modified: June 14, 2004 05:09:37 PM CDT