L3 READOUT

L3 READOUT (non-VBD mode)

• CRATES & SBCS
• CHECK-LIST
• GETTING STARTED
• RUNNING THE SYSTEM
• TROUBLESHOOTING
• MAIN DEBUG COMMANDS
• ADDITIONAL NOTES
• DEBUGGING CTT INPUT PROBLEMS
  

CRATES & SBCS

• x70 (SBC: d0sbc063b)
  
• x72 (SBC: d0sbc054b)
  
• x73 (SBC: d0sbc009b)
  
• x74 (SBC: d0sbc019b)
  
• x75 (SBC: d0sbc004b)
  

CHECK-LIST

• Crates are included in the run (use coormon)
  
• x50 (and x52) are in the run (use Umon)
  
• DAQ is running i.e. there is data flowing to us (you should see a non-zero L3 input rate in Umon for our run)
  
• The FRC is getting input from the CTT data-senders (check the DAV signal: ask an expert if you do not know how to do it)
  
• The SCL cable is connected to the SCL mezzanine card on the FRC (mip1, slot 13)
  
• The crates have all the cards (12 MBs in total) plugged in all the right slots and there is a SBC in slot 4.
  
• Typically, the parallel cable that sends FRC information back to the SCL hub is not plugged in. However, during shifts, this should be done to test if this prevents L3 readout from crashing. However, one has to be very vigilant then cause a STT crash then would bring D0 data-taking to a halt.
  

GETTING STARTED

• Power on the crates (if they are not already powered up).
  The STCs and TFCs are downloaded on power-up.
  
• Log on as d0stt to d0ol36. Go to directory 'stttestpython'
  
• Connect to the IOC of the crate (as described in the STT shifters' page)
  
• Open the download gui: /online/config/stt/utils/py/dwnlGui &
  
• Log on to the SBC as d0runsu:
  ssh d0runsu@d0sbc063b (for Crate x70 for eg.)
  Ask an expert for the password. Once logged in do the following:
  >>> config_x
  c
  use routing
  false
  ^C
  

• Prime the SBC:
  >>>frc_config_test.py stt0 6
  where the first argument specifies the crate and the second the number of test events sent through the system. This particular command sends 6 test events through the system in Crate x70. Do >>>frc_config_test.py stt2 6 for x72 and similarly for other crates.

• To verify the number of events read by the SBC, do:

  >>>getInfo.sh on the SBC window.

  A typical output is here . The line dma_interrupts_OK=6 implies that 6 events have been successfully transferred to the SBC. It's also useful to note that the following are true:
  dma_interrupts_ERR=0
  suspicious_wc_cnt=0
  

RUNNING THE SYSTEM

TROUBLESHOOTING

• First ensure that data is still flowing (the L3 input Rate should be non-zero for our run in Umon).

• Check to see if the L2 part of the system is still working. Do a
  >>> tfc00 status 0.1
  
  and verify that TFC0 is still continuing to get data from the FRC (check that the number of words and blocks for channel 0, mip1 is increasing). Verify the same for TFC1.

  If the number of words/blocks is not increasing, you have to make sure again that there is data-flow and have to check if the TFCs are hung in anyway. If this is the case, both L2 and L3 are hung.

  If the number of words/blocks is increasing, it implies that only L3 is hung. Do the following:

• First stop the FRC by doing >>>frcoff.py stt0 .

• Make a sub-directory to store the debug output. The sub-directory should be located at /scratch/stt/debug/ on d0ol36. For eg. /scratch/stt/debug/0915 where we follow the month-day format for naming the sub-directory. Direct the output of all the following python commands to this area.

• Check the overall FRC and BC status first by doing:

  frc_status.py stt0

  A typical status should look like this . Many of the numbers might be different ofcourse. The important thing to note is if any of the BC status registers shows up as 0xffffffff. This would indicate that the PCI bus on that card (FRC/STC/TFC) has hung and that that is the reason for the crash. There are three cases here:
  (i) The PCI bus on one or more STCs/TFCs has crashed.

  In that case do a mb_reset.py stt0 slot# for that slot in which that particular card/cards sits in. Using the download Gui, choose the option that does not download all the cards on a reboot. And then reboot the cpu by typing in reboot on the IOC terminal.

  Then check the status of that card. Do a python xstc_mon.py stt0 slot# and also do a python STCchk.py stt0 0 if it's a STC or a tfc00 (or tfc01) debug if it's a TFC.

  Reinitialize all the cards.
  >>>tfc00p init allskips daq (or tfc00n init allskips daq depending on the magnet polarity). The same for TFC1.
  >>> python stc_reset_all.py stt0
  >>> frc_config_run.py stt0

  Start the system running again by doing a frcon.py stt0 . If there are still no events getting into the SBC and you've done all the checks listed above again, then the crate might have to be power-cycled.

  (ii) The PCI bus on the FRC is hung.
  

  You may notice that doing a getInfo.sh shows dma_interrupts_ERR=1 . Reset the pci bus on the FRC by doing a mb_reset.py stt0 13 . Then use the download Gui to choose the option that does not download all cards on a reboot. And then reboot the cpu. If the run had still been going on during this period, do a frcoff.py stt0 after the reboot finishes, dump the frc_mon.py information and then reinitialize all cards as above and start again. If L3 readout continues to not work, power-cycle the crate and start fresh again.

  (iii) If none of the PCI busses have crashed, do the following:

  • Check to see if the FRC error and busy bits are set by doing a frc_mon.py stt0 . Save this information for an expert.
  
  
  If L1 ERROR and/or L1 BUSY is true (while L2BUSY is not set), ask for an sclinit and see if that helps. For the sclinit to work the FRC should be on. If you had switched it off earlier, switch it back on by doing a frcon.py stt0 . If this does not help, dump the frc_mon.py information. Check to see if the third (amber) LED on the SCL Mezzanine card is on and make a note of it along with the frc_mon dump. Configure the FRC in run-mode and try again.
  If the L2 ERROR and/or L2 BUSY bits are set, follow the procedure mentioned in the above case.
  If both L1BUSY and L2BUSY are true, follow the steps below.
  • Check to see if the TFC is in any 'hang' mode - STC Read, FRC Read, Fitwait etc.

    If it is a STC Read, follow the instructions in Problems.txt.

    In a nutshell, check the TFC status dump and make a note of which STC (mipsite, channel#) has fewer blocks than the others. Do a status dump for that STC.Do a python xstc_mon.py stt0 slot# and also do a python STCchk.py stt0 0 . Re-initialize all the cards as described above (without rebooting or power-cycling)and start again. If the TFC crashes in STC Read again, and especially if it's the same STC again, power-cycle the crate and start again. If the problem persists with the same STC, ensure that the LVDS cables from that STC and into the corresponding TFC are plugged in properly.

    If the TFC hangs in FRC Read, read out the diagnostic LRB on the FRC by doing:
    >>>frc_lrb_read.py stt0 . The output will be found in the file outputfiles/LRB_output.txt . A typical output is here . Check to see if the first CTT header word (0x2101030d ) is present or no. To fully understand the data format, please look at Hal's T/R document.

    In all TFC non-pci bus hang cases, do a TFC debug dump (there is no need to reboot since the bus is not hung). Ask for an sclinit and start again. If that does not work, reinitialize all cards and start again.

  • Check to see if all the cards are advertising L3 data. Do
    

    FRC: frc_l3_read.py stt0 > l3out.txt .
    A format of the output should look like this . If, instead of this, you see only one word repeated over and over again (for eg. like this ), it implies that there is no data in the FRC L3 fifo.

    STC: python xstc_mon.py stt0 slot# .
    Check the line ADVERTISING L3 DATA : if it's status is yes , it implies that that particular STC has L3 data. Do the same for all STCs.

    TFC: tfc00p status.
    The L3FIFO counter should have a non-zero value if there are events in the TFC's L3 fifo. Do the same for TFC1.

    If none of the 12 cards is advertising L3 data, it might mean that all the cards had their data successfully transferred to the BC and that the crash had happened at a later stage. If all but say one of the 12 cards is not advertising L3 data, then that card is responsible for the L3 readout hang. Please make a note of that.

  • Check the status for all the BCs. Do

    >>>bc_mon.py stt0 slot#

    A typical output looks like this for when there is no problem with the BC.

  • Once you've stored all the debug output, re-initialize all the cards, start the FRC again. If there are still no events, ask the DAQ shifter to send an sclinit. If this does not help either, power-cycle and start fresh again.

  MAIN DEBUG COMMANDS

  And last but not the least, if in any doubt, make sure you store the following outputs for the experts to look at. (And send an email highlighting the problems you saw.)

  Note: If the PCI bus has hung on any MB, you will need to reset the MB first (mb_reset.py), remove the download of all cards from the start-up script and then reboot the cpu before you do the debug for that card. The frc_status.py command should tell you right away if the PCI3 bus on any MB has crashed, so that command should be the first one to use.

  frc_status.py stt0
  frc_mon.py stt0
  tfc00p debug
  tfc01p debug
  python xstc_mon.py stt0 slot# (for all slots)
  python STCchk.py stt0 0
  bc_mon.py stt0 slot# (for all slots)
  frc_lrb_read.py stt0
  python stc_l3_read.py stt0 slot#

  ADDITIONAL NOTES

  • If you want to reset the SBC via software, you can do the following on the SBC window:

    >>>reset_all.sh stop
    >>>reset_all.sh start

    You should prime the SBC again(as described above) if you've reset it.

  • To make sure that we haven't lost sync with the SCL (trigger framework), check that the three LEDs on the SCL mezzanine card on the FRC are on. If only the top two are on, it implies that we have lost sync and in that case we need to configure the FRC in run mode again (by doing frc_config_run.py stt0 for x70 for eg. ) and start again.

  • One can send a maximum of 30 test events through the system by doing frc_config_test.py stt0 30 . For longer-term running use frc_config_test_unlimited.py stt0 . This can be used for debugging the system during periods we get no L1CTT input and for testing hardware etc.

  DEBUGGING CTT INPUT PROBLEMS

  We should try and note which of the following is the problem when we think we have a CTT input problem. This will help the CTT group to debug their firmware and hence provide us more stable inputs.

  (1) BX/TURN mismatch between SCL and CTT
  (2) Missing EOE or BOE
  (3) Totally missing CTT events
  (4) Data corruption
  

  We can get information on the first two problems by doing a frc_mon.py (the top two lines in the dump: BOE/EOE MISSING ERROR, BX/TURN MISMATCH ERROR ). In my experience, a sclinit has almost always solved this sort of problem but we need to gather more statistics.

  The third problem above occurs when the DAV goes missing totally. The typical signature is we go L1 & L2 busy with the following set in frc_mon.py:

  L1_AF_inBM
  L2_AF_inBM

  These, ofcourse, might be set due to STT problems as well, so we have to be careful in reaching the conclusion that it's a CTT problem. Looking at the DAV signal is the more conclusive check. This problem is solved by either a sclinit or a FixCTT depending on what the cause of the problem was in the first place.

  ---

  L3 READOUT (VBD mode)

  Follow all the steps for L3 Readout in non-VBD mode for getting started and running the system and troubleshooting. The parallel cable that sends information back to the SCL hub should always be plugged in. In addition, the following steps have to be taken in order to include the STT crates in VBD mode.

• Ask the DAQ shifter to stop the previous run and include the STT crates in the run via the CRATER GUI.
• Re-download the trigger.
• Do a software re-initialization of all the cards if required. And start the FRC by doing a frcon.py stt0
• Set routing to 'true' for the SBC.
  >>> config_x
  c
  use routing
  true
  ^C
  
• Ask the DAQ shifter then to start the run.
• For taking our crates out of VBD-mode, ask the DAQ shifter to take our crates out of the run and re-download the trigger. Set routing to 'false' for the SBC.




Tulika Bose