Reprocessing: Meeting of 20-Dec-2004: 9:30-10:30 ESNet video conference

Agenda

1. News
2. Status of Implementation of Reprocessing in JIM (update)
3. JIM Deployment and Remote Setup
   
4. Certification of Sites
5. AOB.

Minutes

Participants

Topics

1. News
   Status of p17.02.00 on the farm: Running with floating exceptions turned on leads to a high number of floating point exceptions (one per run). Can hopefully be fixed easily.
2. Status of Implementation of Reprocessing in JIM (update)
   
   • Processing
     Awaiting updated runjob which fixes the bug that large TMBs don't get stored to the durable location and that is able to deal with p17.02.00 which doesn't produce DSTs.
     Also waiting for p17.02.00 tar-ball.
     (Both actually out since Saturday)
   • Merging
     Initial tests successfull. Need to verify sanity checks that prevent duplication of data.
   • Production scripts
     Initial versions of basic scripts done:
     
     • sub_dataset_production
     • sub_production_recovery
     • sub_dataset_merge
     All work with the dataset to the raw input files as their only argument. Need testing, need to allow specification of site and release version.
3. JIM Deployment and Remote Setup
   
   • SAM station setup
     
     • SAM groups
       Due to a limitation in SAM the sum of the cache assigned to the groups at a given site must not exceed the total available cache. Because of this limitations each site needs to dedicate a fixed amount of their cache to the group d0prodruction in order to run production as this group. We recommend number_of_cpus_at_site*1GB.
       If this poses a problem to your site, please let us know asap
     • SAM DB server
       Sites running their own sam_db_server must do
       
       cd $SAM_DB_SERVER_DIR
       cd lib
       (edit) InFile.py
       # look for "def getFileType"
       # in the section with all of the other various thumbnail variants,
       # add 'unmerged-thumbnail' to the list
       
       otherwise the new data_tier "unmerged-thumbnail" will not work for storing.
       
       
     • SAM routing station should be changed to remote-production-router:
       Instructions provided by Robert Illingworth.
   • Status of Remote Sites
     • DØFarm: OK.
       
     • GridKa: Down after moving partitions.
       
     • Lyon:
       
     • SAR:
       
     • WestGrid:
       Downloaded 200(8TB) of 600 projects within 2-3 weeks.
       JIM was running with old station (at computing site)
       Currently moving to new sam station (at storage site)
     • Wisconsin:
       
     • CMS Farm:
       
     • UK:
       Manchester: needs to update JIM and DB Servers.
       
     • Prague:
       
4. Certification of Sites
   Suggestion: Each site processes two daysets with jobfiles_dataset = reco_bin_?? using p17.02.00:
   1. One common to all sites, remote sites must not merge this sample: dayset....
   2. One different for each site. Remote sites should merge this sample after RecoCert was run on the unmerged files.
      • DØFarm: dayset....
        
      • GridKa: dayset....
        
      • Lyon: dayset....
        
      • SAR: dayset....
        
      • WestGrid: dayset....
        
      • Wisconsin: dayset....
        
      • UK:
        dayset....
        
      • CMS Farm: dayset....
        
      • Prague: dayset....
        
   The first set is used to certify the production step by comparing all sites with eachother and with the old farm production method. The second set is used to certify the merging step by comparing the unmerged with the merged plots at each site.
   The need to recertify with p17.03.00 is questioned. Will decide once it is available.
5. AOB.

   Metadata should be checked: Iain says unmerged-thumbnails don't have type. Do we need this? How do we differentiate between physics data and test or certification data? Iain looks into creating a tar-ball p17.02.00-test to avoid interference with real data production.
   Alternatively we can change app_version and app_name in the macro these are only used for metadata. release_name is used to find the software in the tar-ball.
   For this round we'll use an extra tar-ball. For the future we'll provide a test flag in JIM.

   Ursula Basler asks for adding an FBSDBSERVER. Can this be added to reprocessing? Probably yes. Sites need to bring up the corresponding DB server proxxy.

Next Meeting