Reprocessing: Meeting of 22-Nov-2004: 9:30-10:30 ESNet video conference

Agenda

1. News
2. Status of Implementation of Reprocessing in JIM (update)
3. Status of JIM Deployment to Remote Site
4. AOB.

Minutes

Participants

Topics

1. News
   p17.01.00 frozen Monday night. Problems with getting new constants from DB servers. d0reco p17.01 crashes (L1, L2 !?) when configured to read constants from farm server, and creates error messages that indicate it doesn't read the constants at all when configurd to use the users server.

   We don't have a p17.01.00 tarball appropriate for running in JIM.

   It is unclear how long this delays 20pb-1 test.

2. Status of Implementation of Reprocessing in JIM (update)
   
   • Processing:
     Good experience in using fcp in submission. We can achieve a load of nearly 600 within 2 hours (when manually delaying submission). However, we have problems in the reliability of storing to the durable location: Only ~300TMBs and no DST arrived in the durable location. However, 589 of 594 DSTs exist at the head node in the sandbox Still condor polling caused problems. Why doesn't mc_runjob report an error?

     Selecting the DB server via environment variables only overwrites a prefix. The farms serves however differ in a suffix. Can we change the Farms DB servers to fit the naming convention.
     After the meeting Robert Illingworth corrected these statements. The full name of the DB servers can be set with environment variables

   • Merging (Andrew Baranowski) Recoded from scratch: Code to stream data, split data set into chunks to merge, call copyd0m and storing results is done (worker node script). Metadata (about a days work) and envisioned run-time checks are missing. Then also integration into JIM is needed (est. 1 week).
   
   

3. Status of JIM Deployment to Remote Site
   

   Versions to be installed (extended):
   jim_job_managers v2_2_17
   jim_sandbox v2_2_1
   sam_client v1_0_5
   xmldb_client v2_0_4
   vdt v1_1_14_13
   jim_client v2_0_10
   jim_info_provider "current"

   • DØFarm: Working again. Managed to bring load upto to 600 (with fcp hack and delayed submission). Storing to durable location is observed to be unreliable.
     
   • GridKa: 100job test was successfull.
     
   • Imperial: SAM working, JIM packages updated but only basic tests so far. Tests with reprocessing next.
     
   • Lancester: --
     
   • Lyon: JIM upgraded. MC works, Reprocessing has problem with naming service.
     
   • Manchester: --
     
   • Rutherford: --
     
   • SAR: Still investigating McFarm.
     
   • WestGrid: Working on batch adapter. Going to map users to user d0jim (or similar) at head node.
     
   • Wisconsin: 100job test successfull.
     
   
   

4. AOB.

Action Items:

Next Meeting