Hall C DAQ

From HallCWiki
Jump to navigationJump to search

How to kill/reset/restart the DAQ, or, "The DAQ is dead, now what..."

How to connect to VNC session

ssh into cdaq@cdaql1 and from there type: go_nps_daq or go_coin_daq (they are the same).

If running HMS-standalone type go_hms_daq

However, you most likely already know that and have those screens.

If you are connected to the appropriate DAQ vncsession

If you are connected to the appropriate DAQ vncsession and CODA has hung or gotten into a funny state, then bring a terminal window to the foreground and type the appropriate kcoda / startCoda script, the name of which depends on what you are running, which is probably coincidence 

 % killCoin for coincidence DAQ.  (kcoda for NPS-standalone, killHMS for  HMS-standalone.)

Wait for a bit (10 sec). Then restart coda by typing this in the same terminal: 

 % startCoin for coincidence DAQ.  (startCoda for NPS-standalone, startHMS for HMS-standalone.)

A note about the "platform", which is one of the CODA components that runs on the workstation like cdaql6. It should be running 24/7 in accord with "systemd". If you don't see it in the process list (ps -awx | grep -ai platform) then typing "platform &" on coda@cdaql6 (assuming we're running the DAQ on cdaql6) will start it, but make sure you're not running the platform twice. On some rare occassions it may be necessary to restart the platform. In that case, "kill -9" all the platform instances and restart it.

You will then have to complete the 'standard' CODA steps:

  1. Choose 'Platform:Connect' from the CODA Run Control GUI menu bar.
  2. Click on the 'Tools' icon.
  3. Then you should be able to start a run as usual.

If you the components can't connect, it may be that the platform is not running, so you can restart it by typing "platform &" on coda@cdaql6. But first check the process list "ps awx | grep platform"

  • Make an hclog and document the error you saw (screen snapshot, etc) if you choose to do this.

If the kcoda/startcoda process doesn't work

If the system does not restore using the kcoda/startcoda procedure, then you may need to reboot the ROCs following this process 

Hall_C_DAQ#If_you_need_to_reboot_a_ROC.

Be patient, then try the kcoda/startcoda procedure as discussed in that section.


If the CODA vnc session disappeared and 'go_coin_daq' doesn't work

NOTE: Ask a DAQ expert and/or the RC before you do this.  It really should never be needed.

If everything disappeared and 'go_coin_daq' doesn't work then you can bring things up from scratch outside the VNC session. (It usually means the vncserver didn't start on cdaql6, so you could login to coda@cdaql6 and type /home/coda/coda/scripts/startVnc, but that's supposed to happen in cron.)

COIN Mode

This should be our production mode for the experiment. 

 % ssh coda@cdaql6

In that terminal, run: 

 % startCoin

NPS standalone

 % ssh coda@cdaql6

In that terminal, run: 

 % startCoda

HMS standalone

 % ssh coda@cdaql5

In that terminal, run: 

 % startHMS


If the Event Builder (EB) process keeps dying when you try to start a run

This often means there is a memory issue on one of the linux ROCs. It is simplest to follow the Reboot ROCs procedure to resolve this.

How to bring up the DAQ screens on a workstation

The DAQs run inside their own VNC sessions. They can be brought up on any of the Hall C Counting House machines by running these commands logged in as the 'cdaq' user:

go_coin_daq ... coincidence configuration. NPS + HMS crates.

go_hms_daq ... HMS standalone

go_nps_daq ... NPS standalone.

Generally, a crate can only operate with one running DAQ.

If you need to reboot a ROC

If a ROC gets in a really bad state, its crate may need to be power cycled using the procedure and links below. 

It pays to be a little patient with this process.  Stagger turning the crate power back on by 10--20 seconds.
For example, turn on hccrate01, wait 10s, turn on hccrate02, wait 10s, etc...

NOTE: Open a browser window in the CODA VNC session (usually there is one already running in the left-most workspace).
      All of the crate reset links are there along with stored login credentials.

COIN mode

Coincidence (COIN) mode uses both the NPS and HMS ROCs, so you will need to reboot all of them using the links in the two sub-sections below. 

After you have power cycled all the ROCs, wait for 2--3 minutes then run kcodaCoin; and startCoin.  If that doesn't work, call an Expert.
  • Make an hclog when you do this!

A note: SHMS is not used during the NPS experiments.

HMS

The HMS has three ROCs. Follow these links to power-cycle:

  • ROC01 (Counting House Electronics Room)
  • ROC03 (HMS Electronics Hut -- the wire chambers)
  • ROC05 (Counting House Electronics Room) --- On APC switch. See note below.

NPS

The NPS has five ROCs and five VTPs. The VTPs get power cycled when you cycle the ROCs.


Reboot them by clicking on the link, and cycling the power. Wait for 10s after clicking the 'power' button before clicking it again. The page will automatically refresh when it is ready.

After you have power cycled all the ROCs, wait for 2--3 minutes then run the appropriate script (killCoin / startCoin).

ROC5/hccrate05 is on an APC switch. Pick the 'Outlet Status' Tab; click on On to the left of hccrate05 on Outlet 1. Set the Control Action to Reboot Delayed, click the checkbox to the left of hccrate05, then click the Next>> button and click Apply on the next screen. If you know where hccrate05 is in the Counting House Electronics Room, you can power cycle it with the switch on the front panel. Turn it off, wait 5 seconds, and turn it back on.

  • Make an hclog when you do this!

Rebooting the DAQ computers

In rare instances, we have to reboot the cdaql5 (HMS) and/or the cdaql6 (coinc or NPS) computers. This is best done under the guidance of the DAQ expert, but the link to the general procedure is documented here: https://logbooks.jlab.org/entry/4123883

Live Time Display

In should be visible on rcgui. There is tab in the lower right area of the GUI.

Setting Prescales

Prescales.png

Run the prescale GUI using one of these commands from a terminal in the coincidence DAQ session. Do this from a terminal in the appropriate CODA VNC session. 

 % go_prescale_COIN

How to pick a prescale value

The relationship between the prescale setting and the actual prescale value is somewhat complicated. You can click on the 'Help' button for details.

  • The best way to do it is to type in the prescaled rate you want into the Target Rate field and hit return. The Prescaled Rate field will update to show you the rate the DAQ will be accepting.
    • You can try adding or subtracting '1' to the 'Value' field if you want to try to tweak the Prescaled Rate a bit.
 NOTE: You MUST hit SAVE *before* starting the run!

If the scaler rates in the prescale GUI are not updating

This can happen after CODA gets in a funny state, for example.

  • Quit the prescale GUI (Exit button)
  • Run the GUI again (as above).

Some expert notes

Most shifter workers may not need these notes

A note about the vncserver. It may happen that vncserver didn't start on cdaql6. I think I fixed this, but it's easy to check. Login to coda@cdaql6 and type "ps axw | grep vnc". If it's not running, type /home/coda/coda/scripts/startVnc.

What if cdaql6 goes bad ? We should still be able to run on cdaql5 or another cdaq. Just ensure that the enviroment is valid (see below). There could be some remnant in the log-in scripts or in coda scripts which relies on cdaql6; ideally not. 

[coda@cdaql6 ~]$  echo $EXPID
NPS

[coda@cdaql6 ~]$  echo $SESSION
NPS

and these are currently "HMS" on cdaql5, which is used for HMS standalone DAQ.

A note about the "platform". The platform is one of the CODA components that runs on the workstation like cdaql6. It should be running 24/7 in accord with "systemd". If you don't see it in the process list (ps -awx | grep -ai platform) then typing "platform &" on coda@cdaql6 (assuming we're running the DAQ on cdaql6) will start it, but make sure you're not running the platform twice. On some rare occassions it may be necessary to restart the platform. In that case, "kill -9" all the platform instances and restart it.

EDTM Settings

Remotely adjustable EDTM

Currently set with a a command-line helper go_edtm

  • There is a command-line helper go_edtm to set/readback this PV. (the prescale_GUI does not set it at the moment.)
  • More details are in this log entry: EDTM pulser switched to cdaqpi1 output
    • The RPi that controls this is cdaqpi1 which runs an EPICS softIOC.


F250 Pedestal Updates

Overview

This How-To discusses the necessary steps required in order to take a pedestal run utilizing the single arm DAQ's. Furthermore, the steps required in-order to determine the appropriate FADC thresholds are discussed. 

NOTE: This is for EXPERTS ONLY.  If this is done poorly and not caught the FADCs will not work correctly and the data maybe garbage!

Taking Pedestal Runs

  • Ensure that both single arm DAQ's are configured with the "fadcnothr" flag, which is activated via. of the pre-scale GUI, and the configuration has been saved
    • The FADC's can either be in mode 9 or 10
  • With both arms take at least 1000 cosmic or EDTM triggers and note the run numbers

Determining the FADC thresholds

  • The following example will illustrate the necessary steps required to determine the FADC thresholds for SHMS ROC 2
  • On the SHMS side, this procedure will have to be repeated for both ROC's 2 and 4
  • On the HMS side, this procedure only has to be executed for ROC 1
  • With the pedestal runs complete, open a terminal and SSH into cdaql1
ssh cdaq@cdaql1
  • Setup the hallc-online analyzer
go_analysis
  • Descend into the hallc-online hcana/podd directory
cd ../hcana/podd/hana_decode/apps
  • Exectue the tstfadc_main.C script
./tstfadc
  • Input the appropriate configurations: raw data file location, run number, number of events, spectrometer name, crate number
Enter the location of the raw CODA file (raw, raw.copiedtotape): raw
Enter a Run Number (-1 to exit): 1537 
Enter Number of Events to Analyze (-1 = All): -1
All Events in Run 1537 Will be Analayzed
Enter the Spectrometer Name (hms or shms): shms
Enter the Crate Number to be Analyzed: 2
  • If successful, the for file "fadc_data.root" will have been produced
  • In order to produce the threshold text file "thresholds.dat" one needs to execute the calc_thresh_hallc.C ROOT script
root -l ./calc_thresh_hallc.cxx
  • Input the appropriate configurations: spectrometer name, crate number, FADC mode, threshold in channels
Enter the Spectrometer Name (hms or shms): shms
Enter the Crate Number to be Analyzed: 2
Enter The Mode of The F250's: 9
Enter The Number of Channels Above Baseline the Threshold Should Be [40 (10 mV) recommended]: 40
  • A ROOT file "fadc_ped_data.root" will have been created which contains the final fits and determines the baseline values and thus the threshold values which are written to the text file "thresholds.dat"
  • Open the "thresholds.dat" file and have a sanity check the thresholds
    • Most FADC channels should have thresholds set to 400-700 channels
    • For channels set to 4095, these channels are empty
    • For ROC2 slot 14 channels 12-15 are set to 1 by default since they are fast raster FADC channels
    • For ROC4 slot 18 channels 12-15 are set to 1 by default since they are fast raster FADC channels
  • In the event that a few thresholds are set to unrealistic values, simply open the "fadc_ped_data.root" file and perform the fit manually if possible
    • If a manual fit is not possible, then a new pedestal run with more statistics is recommended

Loading the FADC Thresholds

  • These instructions only pertain to the HMS at the moment. For NPS, it is different.
  • In order to have the CRL's load the appropriate FADC threshold files the newly created "thresholds.dat" file must be placed in the correct location
  • SSH into cdaql1 as coda
ssh coda@cdaql1
  • Descend into the respective thresholds directory for the appropriate ROC
cd coda/config_files/SHMS/ROC02/thresholds
  • Copy the newly created "thresholds.dat" into the current working directory and tag the "thresholds.dat" file with the current date
cp hana_decode/thresholds.dat thresholds_DDMONYYYY.dat
  • Ascend one directory and symbolically link the newly created thresholds file as "thresholds.dat"
cd ../
ln -fs thresholds/thresholds_DDMONYYYY.dat thresholds.dat
Retrieved from "https://hallcweb-2024.jlab.org/wiki/index.php?title=Hall_C_DAQ&oldid=22036"