d0repro - Production scripts for p17 data reprocessing
d0repro - Production scripts for p17 data reprocessing
setup upd
upd install -G-c d0repro -h www-d0.fnal.gov
To configure please follow the description in the INSTALL_NOTE whose path is mentioned at the end of the installation. The configuration will remain valid between updates from v0_7_4 onward.
sub_production.py <dataset> <d0release> [--test]
will submit the d0reco production for the given datasets of raw files and mark the production as RUNNING.
It can be used for initial submission as well as for recovery jobs.
Files already available will be automatically excluded.
Please allow 12h after a previous production grid-job for the same dataset finished before resubmitting.
sub_merge.py <dataset> <d0release> [--test] [--nostore]
will submit the merge stop for the files produced from the given datasets of raw files and mark the merge procedure as RUNNING.
It is save to use the same command for recovery if no other merge job for the same dataset is still running.
Please allow 12h after the production grid-jobs are finished before submitting a merge job.
Also allow 12h before reissuing the same command to recover from failures.
check_production.py <dataset> <d0release> [--test]
Will print the current status of the d0reco production for a given dataset of raw files
and store the status of the corresponding production (in production.status).
The status will be set to RUNNING if a running or idle grid job is found.
Else it will be set to COMPLETE if for all files of the input raw dataset an unmerged thumbnail was produced;
the status will be set to PARTIAL if any unmerged thumbnails are missing in SAM and no job is running.
check_merge.py <dataset> <d0release> [--test]
Will print the current status of the d0reco production and merging for a given dataset of raw files.
and store the status of the corresponding production and merge steps (in production.status and merge.status).
undeclare_ghosts.py <dataset> <d0release> [--test] [--force]
Will search for merged files declared to sam, but without location
and for unmerged-thumbnails declared to sam without location that don't have a merged daughter.
After a 10s waiting time in which the script might be aborted with Ctrl-C it tries to
"sam undeclare" these files.
All grid-jobs must be completed since more than 12hours for the command to operate.
--force will lift this requirement.
clean_completed.py
will move projects that has a merge status of complete or finished (from comlete of finished inputs) to a subdirectory.
check_all.py
This command will run do check_merge.py on all jobs currently available in the d0repro-work directory.
list_status.py [--all]
This will summarise the status of all jobs as computed by a previously run check_all.py or check_merge.py.
For jobs which are in status PARTIAL the auto-pilot suggestion is given.
--all Also jobs in status NEW are listed. By default these are suppresed.
auto_pilot.py [ --all | --merge-only | --production-only | --auto-approve]
This will summarise the status of all jobs as computed by a previously run dcheck_all.pyd/check_merge.py
For jobs which are in status PARTIAL the auto-pilot suggestion is given (as list_status.py).
In addition a script named "Autopilot.sh" is created in the d0repro-work directory.
--all - jobs in status NEW are listed. By default these are suppresed.
--merge-only - only sub_merge commands will by added to the script Autopilot.sh
--production-only - only sub_production commands will by added to the script Autopilot.sh
--auto-approve - in addition to the sub_* commands some datasets which are in status NEW will be moved to status APPROVED.
The number of datasets to be approved is currently identical to the number of merge jobs suggested by the autopilot.