(r10) AlpgenProduction31X < Main

Tags: , view all tags

---+ ALPGEN Production in 31X

---++ Goals
   * Test the Alpgen Prouduction Workflow in CMSSW 3_1_X

---++ Modus Operandi
   1. Creation of ASCII LHE files outside the CMS framework
   1. Publication of corresponding articles into MCDB (http://mcdb.cern.ch)
   1. Execution of CMSSW jobs (at CERN)for retrieving the files, translating the ASCII LHE files into EDM LHE files and publishing them to DBS
   1. Deployment of the files to the T2s for production.

   * The responsabilty of the Generator group is for steps 1 and 2. DMWM will execute the jobs for accomplishing 3 and 4.
   * Preferred file size from !DataOps: 1-2 GB, ~ 1M events in the same LHE file.

---++ Action Items
   * Try to generate large (1M) LHE fils and upload to MCDB.
      * 50 jobs of 2000000 events each, ttbar+1 jet, 40 !GeV, should produce some 900K ttbar+1j unw events. Submitted to Condor. 
      * Each job takes in average 3min 15sec!
      * Efficiency: 0.91% (average 18200 events obtained per job)
      * If I just concatenate the .unw files, !AlpgenSource does not care... does it? No it doesn't =D

-- Main.ThiagoTomei - 18 May 2009

   * Questions:
   1. What to do with the cross-sections and the luminosity, when concatening different files?
      * Sum the luminosities, average the cross-sections, average/sqrt(N) the errors?
   1. What is the workflow? For minimum startup conditions:
      a. ALPGEN Standalone + Condor --> generate many .wgt files, unweight them to many .unw files. Use the 1st script below for this.
      a. Script to concatenate all .unw files, write a single, unified _unw.par file, and write a .tar.gz file containing ALL the .wgt and .par files. Use the 2nd script below for this.
      a. cmsRun jobs + LHEWriter --> generate a single, large .lhe file. *%RED%Needs CMSSW 3_1_X%ENDCOLOR%*
      a. Write the !HepML header. *%RED%Needs !HepML working%ENDCOLOR%*
      a. upload to MCDB --> there is a Perl script from http://mcdb.cern.ch for that, but does it work from outside CERN? It uses SRM to access CASTOR.
      a. Coach somebody to test the following parts (MCDBSource, split and merge).
-- Main.ThiagoTomei - 19 May 2009

   1. UPDATE: managed to run the full chain up MCDB source (article 239).
      * But there is a problem, CMSSW_3_1_X is not yet installed at SPRACE.
   1. UPDATE: things that should be in the test suite:
      * Check if grids are correct (maybe not needed now, for we are using LHE files?)
      * Check if split and merge works.

-- Main.ThiagoTomei - 22 May 2009

Results from Marco (AlpgenTimeAndSizeStudies):
   * Running time is linear with number of events requested.
   * Running time is exponential with number of extra jets.
   * In worst case (ttbar + 4jets), 10M events runs on ~ 1h30min in a Phase III node. This is a good turnaround time.
   * I think that planning the production in chunks of 10M events is reasonable. 
      * ttbar + 4jets case: from AlpgenGridProductionPage, one can say that the unweighting efficiency is ~ 5E-04. That, compound with a matching efficiency of 0.15, means that to get 100K events in the end one should ask for ~ 1.35G events. That translates into 135 jobs, a reasonable number.
      * ttbar + 0jets case: from AlpgenGridProductionPage, one can say that the unweighting efficiency is ~ 2E-02, matching efficiency is ~ 0.7. To get 100K events in the end, one needs to ask for ~ 7.1M events. Would rather ask for 100M, divided in 10 jobs, and aim for 1M events in the end.
      * *CAVEAT*: for ttbar + 4jets, 100K events is plenty, but for ttbar + 0 jets it is almost nothing. The two factors (ease of producing X number needed) compensate to a degree.
   * What would I quote as ''full farm capacity'':
      * 116 Phase III Worker Nodes.
      * Worst case scenario: ttbar + 4jets, 6.6M asked events / hour --> 500 UNW events / hour.
      * Total: 58000 ttbar + 4jets  events / hour.

-- Main.ThiagoTomei - 05 Jun 2009

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
txt	sprace_ALPGEN_merge.py.txt	r1	manage	3.0 K	2009-06-05 - 08:07	ThiagoTomei	Script to merge the files produced by the sprace_ALPGEN_submit.py script. It produces a merged .unw file, a coherent _unw.par file, and a zipped directory contating the .wgt and .par files.
txt	sprace_ALPGEN_submit.py.txt	r1	manage	3.3 K	2009-06-03 - 15:24	ThiagoTomei	Script to submit many copies of the same Alpgen generation job to CONDOR. Automatically takes care of random seeds. Useful wiwh seq \| xargs command.

Topic revision: r10 - 2009-06-05 - ThiagoTomei

antalya escort bursa escort eskisehir escort istanbul escort izmir escort