For a given Rivet version, we want to be able to document which analysis/generator combination is known to work (and how to run it). A framework should be created that makes it as easy as possible to generate such a validation table, in the beginning exemplifying it for e.g. two analyses and two generators.
| | Gen1 | Gen2
|
| Ana1 | working | not tested
|
| Ana2 | not working | working
|
Per analysis (linked from each row title):
- Doxygen documentation similar to Debrecen tutorial
Per generator (linked from each column title):
- Standardized script to run the generator (assuming given parameter files)
Per analysis/generator combination (linked from each cell):
- Parameter files (also for multiple runs per analysis)
- Histogram files
- Plot page
Most of the work is probably in the parameter files for each A/G and in figuring out a way to determine working/not working (maybe with respect to a previous version of Rivet or, more difficultly, the generator). The latter might not need to be automatic at first.
There will be two separate phases implemented in separate "scripts", probably in Python:
- Distributing the jobs to the GRID/batch system/...: This will operate on a column (optionally with analysis-exclusion) and start a daemon in the background which knows about all jobs submitted and their status. It would be good to have it persistable, such that it can be restarted in a certain state, should it get killed accidently (cf. Pickle).
- Analysing the output of jobs: Plots, comparison RivetA vs. RivetB or GenA vs. GenB etc., creating the matrix, ...
The distribution tool needs to be able to run essentially arbitrary scripts for each generator, to allow optimisations like caching Sherpa libraries and integration results, or running with sockets/FIFOs rather than AGILe. The Rivet configuration and analysis merging parts can be less general since the interface is constant.
The distribution tool should be written so as not to preclude integration of validation tests like this into a continuous build system like Hudson (used by Herwig++): https://hudson.dev.java.net/
