Analysis Tools Meeting 10/1/2002 Tasks for implementing Two-Stream Streaming. I'll just go down the list... 1) _Supervisor fixes__ We're still waiting on Gordon to fix the Supervisor so it'll pass the streaming XML on to level 3 "quoted" properly. I heard from Gordon over the weekend that it's on his list to do. I sent another nagging e-mail to him today (stressing the importance of this hopefully easy task). We can't do any testing without it. 2) __Test Level 3 streaming online__ Can't do it without 1) 3) __Stream Scheme database__ Jeremy is nearly done. 4) __Setup SAM autodestination__ Done. Streams will be called, stream1, stream2, and monitor 5) __Does SAM store the physical stream correctly__ We think so. Jeremy and I have to look again. 6) __Will the Mark and Pass stream (monitor) blow up SAM__ So long as the monitor stream doesn't make an event catalog, SAM will be fine (cuz monitor is an inclusive stream and the event catalog can't handle duplicated events). We've been told that datalogger does not save a catalog for any trigger with monitor in the name. -- Actually, the datalogger does write an event catalog for the monitor stream, but dlcat chooses not to send it to SAM (but it does get sent to the luminosity system). 7) __Who keeps the list of valid stream names__ SAM does. 8) __Update DB server code with stream override stuff__ Adam has coded this up and tested it. We'll wait for Elizabeth before putting it in production (she may have other changes too). If this goes in late, it's not a big problem since it's really only needed for the web entry and reporting of the trigger database, and we need to change that too. But we can put things in the database by hand for now. 9) __Stream version written by core to BRUNS file__ Scott is working on this (got mail from him today). I thought that coor should also write out the list of streams to the BRUN file -- turns out it already does this. 10) __Luminosity integrity checks online__ See below. 11) __Test physical stream dimension__ Would like to make this part of 2) above. 12) __Put p12 on the L3 Farm__ Jon has fixed many bugs and done extensive testing offline, but some filters still need certification. Problems could delay deployment of p12. 13) __Put stream id into L3 chunk__ Jon has done this, but notes that reco tends to throw this information away for some unknown reason. 14) __Update trigger DB entry/report web pages__ Will wait for Elizabeth to return. No great hurry on this. 15) __Stream XML generator__ Adam will start working on this. 16) __Add inverse stream map to RUNS database__ Adam must write an example of this for Jeremy -- the rest are either done or not needed for Oct7 -- Luminosity -- As of 10/1: Latest hitch are the 000 partition files. Events in 000 partitions are those that come in "unexpectedly" by the L3 farm. They can be evens that show up after the run is closed, or when a farm node crashes -- the events waiting for that node will be sent unprocessed by L3 to the 000 partition. The problem is that luminosity blocks will be repeated in the 000 partitions, violating Michael's integrity checks. Michael has agreed to ignore the 000 partitions in so far as the unitarity checks. Luminosity -- As of 10/2: Michael has realized that the offline luminosity system is insufficient. If someone uses the SAM event catalog to get only the files that have your desired trigger, you will only get the luminosity reported by blocks in those files. If your trigger and stream were rare, you'll miss other blocks in the run and so you'll get a low incorrect luminosity. To fix this means a complete rethinking of the offline (and some online) luminosity reporting system. Furthermore, if the system sees a luminosity block processed by two different versions of RECO (which could happen even in two-streams), he'll flag that block as bad. So all events reprocessed will be flagged as bad. Reports from the physics group: Breese, Reinhold and Brian were present from the top group (thanks guys!): Some misunderstanding on the use of "Monitor stream and Mark and pass events". Reinhold had requested that Mark and Pass events go to the express stream. Remember that "monitor triggers" are physics triggers. They go to the exclusive physics streams. Mark and Pass are special events which go inclusively to the "monitor STREAM". M&P events need to be rare (since their event size is large). The monitor stream should be < 5% and probably will not be easily analyzed by the physics groups. So no physics in the monitor stream! Just L3 testing. The top group has identified tight signal triggers to go into the express stream -- they make up 5 Hz. They've also identified background and monitor triggers that make up another 1.5 Hz (rates measured for 2e31). There's Boaz's 10% right there. The 5Hz includes e+jets, mu+jets, di-e, di-mu, e-mu. All jets are not in the express stream. Heidi has the W/Z official statement -- they see no gain from a 10% stream. 30% may be useful. W/Z will just use first pass processing, but top and higgs don't like that cuz they would like to use the W/Z events for their backgrounds. Perhaps a better solution is to have a stream big enough for useful high Et e's and mu's for the W/Z group that top and higgs could use as well. If you don't get to reprocess it all for Moriond, life goes on. It's a trade off between luminosity and efficiency.