Online Verification

Andrzej Zieminski (.html and comments by James T. Linnemann)

Aim

The goal was to compare online triggering results with the results of simulations bit by bit for hardware and bank by bank for software triggers

Data

ALL stream (once a month or so)
MONITOR stream
- 2-4% of the input data to L1.5, L2, with contents adjusted as needed)
- run essentially every real data run
- about 500 events/file

Tasks (In order of execution)

L2_COMPARE (L2 bank comparison)
AND_OR (L1 Cal input bits)
L1.5 (L1.5 Cal trigger)
MUON (L1, L1.5 muon trigger inputs)

Comment: The tasks were run as separate executables rather than a single executable so easier to link and modify, and so that higher trigger level diagnosis could proceed even in face of lower level trigger problems

Outputs (example from L2 compare)

Summary for individual trigger bits and for L2 banks
details of any discrepancies found
STA's for bad events for future inspection

Statistical summaries

  JAUX BANK processed     : 13513
  JAUX BANK discrepancies :     0
  L2EM BANK processed     : 11258 
  L2EM BANK discrepancies :     1
  PNUT BANK processed     :  4951
  PNUT BANK discrepancies :     0
  MUOT BANK processed     :  5852
  MUOT BANK discrepancies :   182
  LETA BANK processed     :  4396
  LETA BANK discrepancies :    35
  TOTAL events compared   : 26882

Successes

L2_compare
- found several minor problems, particularly bug in treatment of COARSE HADRONIC cells
- gave extra protection in case of mistakes
- capable of finding only
  - mismatches between online and simulation environment (eg ELN vs VMS runtime library) since used same .OLB's on same kind of VAX hardware
  - history-dependent bugs (but nearly the only way to find them!)
AND_OR several minor hardware problems corrected
L1.5 see comments by Dan Owen
MUON
- Essential for correcting trigger problems
- found such problems ~ every 2 weeks

Glitches

delays in availability of TRIGGER files in archive from DAQ experts
disk space and
- always on the ragged edge
- 1.5-2 M Blocks used
- shortage of CPU caused backlogs: need dedicated machines
- needed space for 6 run's worth of data (up to 1/2 or 1 day)
procedural delays in getting latest release of L2 simulation
negligence (once 3 days without verification)
SHADOW running confused L2_COMPARE

Suggested Changes

checking could be part of GM shift responsibility
frequency of running probably about right
5-10 M Blocks of disk
dedicated CPU for these tasks
need designated experts to chase problems (eg the MUOT discrepancies in L2 which were never chased down)
better Documentation for each of the separate tasks