Difference between revisions of "Hall C DAQ"

From HallCWiki
Jump to: navigation, search
(Switching Back to Single Arm)
 
(27 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= How to bring up the DAQ =
+
= How to kill/reset/restart the DAQ, or, "The DAQ is dead, now what..." =
 +
 
 +
=== If you are connected to the appropriate DAQ vncsession ===
 +
 
 +
If you are connected to the appropriate DAQ vncsession (''see below''), and
 +
CODA has hung, or gotten into a funny state, then bring the terminal window at
 +
lower left to the foreground and type the following:
 +
  % kcoda
 +
Wait for a bit (10 sec).  Then restart coda by typing this in the same terminal:
 +
  % startcoda
 +
 
 +
You will then have to complete the 'standard' CODA steps:
 +
# Choose 'Platform:Connect' from the CODA Run Control GUI menu bar.
 +
# Click on the 'Tools' icon.
 +
# Then you should be able to start a run as usual.
 +
 
 +
* ''Make an hclog and document the error you saw (screen snapshot, etc) if you choose to do this.''
 +
 
 +
=== If the kcoda/startcoda process doesn't work ===
 +
 
 +
If the system does not restore using the kcoda/startcoda procedure, then you may need to reboot the ROCs
 +
following this process
 +
[[Hall_C_DAQ#If_you_need_to_reboot_a_ROC]].
 +
Be patient, then try the kcoda/startcoda procedure as discussed in that section.
 +
 
 +
 
 +
=== If the CODA vnc session disappeared and 'go_shms_daq/go_hms_daq' doesn't work ===
 +
 
 +
NOTE: Ask a DAQ expert and/or the RC '''before''' you do this.  It really should never be needed.
 +
 
 +
If everything disappeared and 'go_shms_daq/go_hms_daq' doesn't work then you
 +
can bring things up from scratch outside the VNC session.
 +
 
 +
==== SHMS or COIN Mode ====
 +
  % ssh coda@cdaql6
 +
In that terminal, run:
 +
  % startcoda
 +
 
 +
==== HMS ====
 +
  % ssh coda@cdaql5
 +
In that terminal, run:
 +
  % startcoda
 +
 
 +
=== NOTES ===
 +
==== 'udpSend write failure' warnings are generally '''not''' a problem (ROC5, ROC6, ROC8) ====
 +
 
 +
This is generally a temporary error associated with a scaler process in the ROCs metioned.  It will reset itself once CODA is running again.
 +
''Note that you should '''not''' see this error in the middle of an ongoing run however.''
 +
 
 +
==== If the Event Builder (EB) process keeps dying when you try to start a run ====
 +
 
 +
This often means there is a memory issue on one of the linux ROCs.  It is simplest to follow the [[Hall_C_DAQ#If_you_need_to_reboot_a_ROC | Reboot ROCs]] procedure to resolve this.
 +
 
 +
= How to bring up the DAQ screens on a workstation =
 
The DAQs run inside their own VNC sessions.  They can be brought up on any
 
The DAQs run inside their own VNC sessions.  They can be brought up on any
 
of the Hall C Counting House machines by running these commands logged in
 
of the Hall C Counting House machines by running these commands logged in
Line 13: Line 66:
 
==== HMS Single Arm ====
 
==== HMS Single Arm ====
 
   % go_hms_daq
 
   % go_hms_daq
* The following run parameters should be the defaults for SHMS single arm running:
+
* The following run parameters should be the defaults for HMS single arm running:
 
** Expid:  HMS
 
** Expid:  HMS
 
** Session: HMS
 
** Session: HMS
** Configuration: SHMS_all
+
** Configuration: HMS_all
  
==== COINCIDENCE Mode ====
+
==== COIN Mode ====
===== Setup =====
+
  % go_shms_daq '''OR''' go_coin_daq (they do the same thing)
 +
* The following run parameters should be the defaults for COIN mode running:
 +
** Expid:  SHMS
 +
** Session: SHMS
 +
** Configuration: COIN_all
 +
 
 +
==== Switching between COINCIDENCE Mode and Single Arm DAQs ====
 +
===== Switching to Coincidence Mode from the Single Arm configuration =====
 
Coincidence mode basically adds the HMS crates to the SHMS DAQ.  You should first shut down both single arm DAQs
 
Coincidence mode basically adds the HMS crates to the SHMS DAQ.  You should first shut down both single arm DAQs
 
   % kcoda  # In SHMS DAQ session
 
   % kcoda  # In SHMS DAQ session
Line 27: Line 87:
 
   % go_COIN_DAQ_Enable
 
   % go_COIN_DAQ_Enable
 
You will be prompted to reboot/power-cycle the HMS crates.  Do so following
 
You will be prompted to reboot/power-cycle the HMS crates.  Do so following
[[Hall_C_DAQ#If_you_need_to_reboot_a_ROC]].  Then restart the DAQ in the SHMS DAQ session:
+
[[Hall_C_DAQ#If_you_need_to_reboot_a_ROC]].  Then restart the DAQ in the '''SHMS''' DAQ session:
 
   % startcoda
 
   % startcoda
 
You should see more terminals pop up on the right associated with the additional DAQ components.
 
You should see more terminals pop up on the right associated with the additional DAQ components.
Line 37: Line 97:
 
** Configuration: COIN_all
 
** Configuration: COIN_all
  
===== Switching Back to Single Arm =====
+
===== Switching Back to Single Arm DAQs =====
 
   % kcoda
 
   % kcoda
 
   % go_COIN_DAQ_Disable
 
   % go_COIN_DAQ_Disable
Line 45: Line 105:
 
You should see more terminals pop up on the right associated with the additional DAQ components.
 
You should see more terminals pop up on the right associated with the additional DAQ components.
  
Start up the single Arm DAQs as usual.
+
Start up the single Arm DAQs as usual. You will need to restore the single-arm CODA session type for the SHMS DAQ.  See [[Hall_C_DAQ#SHMS_Single_Arm]]
  
 
If the system is being fussy.  Reboot ''ALL'' of the crates following [[Hall_C_DAQ#If_you_need_to_reboot_a_ROC]].  Then try ''startcoda'' again.
 
If the system is being fussy.  Reboot ''ALL'' of the crates following [[Hall_C_DAQ#If_you_need_to_reboot_a_ROC]].  Then try ''startcoda'' again.
  
= How to kill/reset/restart the DAQ =
+
= If you need to reboot a ROC =
=== If you are connected to the appropriate DAQ vncsession ===
+
  
If you are connected to the appropriate DAQ vncsession (''see above''), and
+
If a ROC gets in a really bad state, its crate may need to be power cycled using the procedure and links below.
CODA has hung, or gotten into a funny state, then bring the terminal window at
+
lower left to the foreground and type the following:
+
  % kcoda
+
Wait for a bit (10 sec). Most of the coda windows should disappear.  (The
+
VxWorks ROC consoles may stay up.). Then restart coda by typing this in the
+
same terminal:
+
  % startcoda
+
  
* ''Make an hclog when you do this!''
+
It pays to be a little patient with this process.  Stagger turning the crate power back on by 10--20 seconds.
 +
For example, turn on hccrate01, wait 10s, turn on hccrate02, wait 10s, etc...
  
=== If everything disappeared and 'go_shms_daq/go_hms_daq' doesn't work ===
+
==== COIN mode ====
 +
Coincidence (COIN) mode uses ''both'' the '''SHMS''' and '''HMS''' ROCs, so you will need to reboot '''all''' of them using the links in the two sub-sections below.
  
If everything disappeared and 'go_shms_daq/go_hms_daq' doesn't work then you
+
After you have power cycled all the ROCs, wait for 2--3 minutes then run '''kcoda'''; and '''startcoda'''.  If that doesn't work, call an Expert.
can bring things up from scratch outside the VNC session.
+
 
+
==== SHMS ====
+
  % ssh coda@cdaql4
+
In that terminal, run:
+
  % startcoda
+
 
+
==== HMS ====
+
  % ssh coda@cdaql5
+
In that terminal, run:
+
  % startcoda
+
  
 
* ''Make an hclog when you do this!''
 
* ''Make an hclog when you do this!''
 
= If you need to reboot a ROC =
 
  
 
==== SHMS ====
 
==== SHMS ====
Line 92: Line 133:
 
Reboot them by clicking on the link, and cycling the power.  '''Wait for 10s after clicking the 'power' button ''before'' clicking it again'''.  The page will automatically refresh when it is ready.
 
Reboot them by clicking on the link, and cycling the power.  '''Wait for 10s after clicking the 'power' button ''before'' clicking it again'''.  The page will automatically refresh when it is ready.
  
  If a ROC console does not show up for you, then run '''kcoda'''; wait for 3 minutes; then '''startcoda'''.  If that doesn't work, call an Expert.
+
  After you have power cycled all the ROCs, wait for 2--3 minutes then run '''kcoda'''; and '''startcoda'''.  If that doesn't work, call an Expert.
  
 
* ''Make an hclog when you do this!''
 
* ''Make an hclog when you do this!''
Line 107: Line 148:
 
back on.
 
back on.
  
  If a ROC console does not show up for you, then run '''kcoda'''; wait for 3 minutes; then '''startcoda'''.  If that doesn't work, call an Expert.
+
After you have power cycled all the ROCs, wait for 2--3 minutes then run '''kcoda'''; and '''startcoda'''.  If that doesn't work, call an Expert.
  
 
* ''Make an hclog when you do this!''
 
* ''Make an hclog when you do this!''
Line 114: Line 155:
 
[[Image:livetime.png|thumb]]
 
[[Image:livetime.png|thumb]]
 
If it isn't showing up (ie. after a kcoda/startcoda) then:
 
If it isn't showing up (ie. after a kcoda/startcoda) then:
 
+
* Select the 'Live Time' tab on the right side of the GUI below the 'Run Status' box.
* Click on 'TS0x' in component list under 'Name' column on left of rcGUI
+
* Click on 'TS0x' in component list under 'Name' column on left of rcGUI to select that component.
* Select Options:Chart:Add
+
* With 'TS0x' selected at the top of the GUI select Options:Charts:Add.
* Seelct 'Live Time' tab
+
* The live time of 'TS0x' should now plot in the 'Live Time' tab.
  
 
= Setting Prescales =
 
= Setting Prescales =
Line 123: Line 164:
 
Run the prescale GUI using one of these commands from a terminal in the SHMS or HMS DAQ session:
 
Run the prescale GUI using one of these commands from a terminal in the SHMS or HMS DAQ session:
  
   % go_prescales_SHMS   -- OR -- go_prescales_HMS
+
   % go_prescales_SHMS -- OR -- go_prescales_HMS -- OR -- go_prescales_COIN
  
 
== How to pick a prescale value ==
 
== How to pick a prescale value ==
Line 132: Line 173:
  
 
   NOTE: ''You MUST hit '''SAVE''' *before* starting the run!''
 
   NOTE: ''You MUST hit '''SAVE''' *before* starting the run!''
 +
 +
== If the scaler rates in the prescale GUI are not updating ==
 +
 +
This can happen after CODA gets in a funny state, for example.
 +
 +
* Quit the prescale GUI (Exit button)
 +
* Run the GUI again (as above).
 +
 +
= EDTM Settings =
 +
{{:Hall_C_EDTM_Pulser}}
 +
 +
= F250 Pedestal Updates =
 +
{{:Hall_C_Pedestal_Runs_and_Determining_FADC_Thresholds}}

Latest revision as of 18:42, 29 August 2021

How to kill/reset/restart the DAQ, or, "The DAQ is dead, now what..."

If you are connected to the appropriate DAQ vncsession

If you are connected to the appropriate DAQ vncsession (see below), and CODA has hung, or gotten into a funny state, then bring the terminal window at lower left to the foreground and type the following:

 % kcoda

Wait for a bit (10 sec). Then restart coda by typing this in the same terminal:

 % startcoda

You will then have to complete the 'standard' CODA steps:

  1. Choose 'Platform:Connect' from the CODA Run Control GUI menu bar.
  2. Click on the 'Tools' icon.
  3. Then you should be able to start a run as usual.
  • Make an hclog and document the error you saw (screen snapshot, etc) if you choose to do this.

If the kcoda/startcoda process doesn't work

If the system does not restore using the kcoda/startcoda procedure, then you may need to reboot the ROCs following this process

Hall_C_DAQ#If_you_need_to_reboot_a_ROC.

Be patient, then try the kcoda/startcoda procedure as discussed in that section.


If the CODA vnc session disappeared and 'go_shms_daq/go_hms_daq' doesn't work

NOTE: Ask a DAQ expert and/or the RC before you do this.  It really should never be needed.

If everything disappeared and 'go_shms_daq/go_hms_daq' doesn't work then you can bring things up from scratch outside the VNC session.

SHMS or COIN Mode

 % ssh coda@cdaql6

In that terminal, run:

 % startcoda

HMS

 % ssh coda@cdaql5

In that terminal, run:

 % startcoda

NOTES

'udpSend write failure' warnings are generally not a problem (ROC5, ROC6, ROC8)

This is generally a temporary error associated with a scaler process in the ROCs metioned. It will reset itself once CODA is running again. Note that you should not see this error in the middle of an ongoing run however.

If the Event Builder (EB) process keeps dying when you try to start a run

This often means there is a memory issue on one of the linux ROCs. It is simplest to follow the Reboot ROCs procedure to resolve this.

How to bring up the DAQ screens on a workstation

The DAQs run inside their own VNC sessions. They can be brought up on any of the Hall C Counting House machines by running these commands logged in as the 'cdaq' user:

SHMS Single Arm

 % go_shms_daq
  • The following run parameters should be the defaults for SHMS single arm running:
    • Expid: SHMS
    • Session: SHMS
    • Configuration: SHMS_all

HMS Single Arm

 % go_hms_daq
  • The following run parameters should be the defaults for HMS single arm running:
    • Expid: HMS
    • Session: HMS
    • Configuration: HMS_all

COIN Mode

 % go_shms_daq OR go_coin_daq (they do the same thing)
  • The following run parameters should be the defaults for COIN mode running:
    • Expid: SHMS
    • Session: SHMS
    • Configuration: COIN_all

Switching between COINCIDENCE Mode and Single Arm DAQs

Switching to Coincidence Mode from the Single Arm configuration

Coincidence mode basically adds the HMS crates to the SHMS DAQ. You should first shut down both single arm DAQs

 % kcoda  # In SHMS DAQ session
 % kcoda  # In  HMS DAQ session

Then switch to COIN mode in the SHMS DAQ session, and start CODA

 % go_COIN_DAQ_Enable

You will be prompted to reboot/power-cycle the HMS crates. Do so following Hall_C_DAQ#If_you_need_to_reboot_a_ROC. Then restart the DAQ in the SHMS DAQ session:

 % startcoda

You should see more terminals pop up on the right associated with the additional DAQ components.

Connect to the Platform as usual, then switch the Session to COIN_all in the Run Control GUI using the Configurations:Sessions menu item.

  • The following run parameters should be the defaults for SHMS+HMS coincidence running:
    • Expid: SHMS
    • Session: SHMS
    • Configuration: COIN_all
Switching Back to Single Arm DAQs
 % kcoda
 % go_COIN_DAQ_Disable

You will be prompted to reboot/power-cycle the HMS crates. Do so following Hall_C_DAQ#If_you_need_to_reboot_a_ROC. Then restart the DAQ in the SHMS DAQ session:

 % startcoda

You should see more terminals pop up on the right associated with the additional DAQ components.

Start up the single Arm DAQs as usual. You will need to restore the single-arm CODA session type for the SHMS DAQ. See Hall_C_DAQ#SHMS_Single_Arm

If the system is being fussy. Reboot ALL of the crates following Hall_C_DAQ#If_you_need_to_reboot_a_ROC. Then try startcoda again.

If you need to reboot a ROC

If a ROC gets in a really bad state, its crate may need to be power cycled using the procedure and links below.

It pays to be a little patient with this process.  Stagger turning the crate power back on by 10--20 seconds.
For example, turn on hccrate01, wait 10s, turn on hccrate02, wait 10s, etc...

COIN mode

Coincidence (COIN) mode uses both the SHMS and HMS ROCs, so you will need to reboot all of them using the links in the two sub-sections below.

After you have power cycled all the ROCs, wait for 2--3 minutes then run kcoda; and startcoda.  If that doesn't work, call an Expert.
  • Make an hclog when you do this!

SHMS

The SHMS has four ROCs. You can contact their crates with these links using a browser from the cdaq account

  • ROC02 (Counting House Electronics Room)
  • ROC04 (SHMS Electronics Hut)
  • ROC06 (SHMS Electronics Hut)
  • ROC08 (Counting House Electronics Room)

Reboot them by clicking on the link, and cycling the power. Wait for 10s after clicking the 'power' button before clicking it again. The page will automatically refresh when it is ready.

After you have power cycled all the ROCs, wait for 2--3 minutes then run kcoda; and startcoda.  If that doesn't work, call an Expert.
  • Make an hclog when you do this!

HMS

The HMS has two ROCs. You can contact their crates with these links:

  • ROC01 (Counting House Electronics Room)
  • ROC03 (SHMS Electronics Hut)
  • ROC05 (Counting House Electronics Room) --- On APC switch. See note below.

The first two work the same as the SHMS crates.

ROC5/hccrate05 is on an APC switch. Pick the 'Outlet Status' Tab; click on On to the left of hccrate05 on Outlet 1. Set the Control Action to Reboot Delayed, click the checkbox to the left of hccrate05, then click the Next>> button and click Apply on the next screen. If you know where hccrate05 is in the Counting House Electronics Room, you can power cycle it with the switch on the front panel. Turn it off, wait 5 seconds, and turn it back on.

After you have power cycled all the ROCs, wait for 2--3 minutes then run kcoda; and startcoda.  If that doesn't work, call an Expert.
  • Make an hclog when you do this!

Live Time Display

Livetime.png

If it isn't showing up (ie. after a kcoda/startcoda) then:

  • Select the 'Live Time' tab on the right side of the GUI below the 'Run Status' box.
  • Click on 'TS0x' in component list under 'Name' column on left of rcGUI to select that component.
  • With 'TS0x' selected at the top of the GUI select Options:Charts:Add.
  • The live time of 'TS0x' should now plot in the 'Live Time' tab.

Setting Prescales

Prescales.png

Run the prescale GUI using one of these commands from a terminal in the SHMS or HMS DAQ session:

 % go_prescales_SHMS -- OR -- go_prescales_HMS -- OR -- go_prescales_COIN

How to pick a prescale value

The relationship between the prescale setting and the actual prescale value is somewhat complicated. You can click on the 'Help' button for details.

  • The best way to do it is to type in the prescaled rate you want into the Target Rate field and hit return. The Prescaled Rate field will update to show you the rate the DAQ will be accepting.
    • You can try adding or subtracting '1' to the 'Value' field if you want to try to tweak the Prescaled Rate a bit.
 NOTE: You MUST hit SAVE *before* starting the run!

If the scaler rates in the prescale GUI are not updating

This can happen after CODA gets in a funny state, for example.

  • Quit the prescale GUI (Exit button)
  • Run the GUI again (as above).

EDTM Settings

NEW (Aug 2021) remotely adjustable EDTM

Generally this will be set using the field in the Prescale GUI, but there is also a command-line helper go_edtm
  • There is a new command-line helper go_edtm to set/readback this PV. It will also be settable in the CODA prescale GUIs.
  • More details are in this log entry: EDTM pulser switched to cdaqpi1 output
    • The RPi that controls this is cdaqpi1 which runs an EPICS softIOC. Ellen B. or Brad S. should be contacted if that device has an issue.

Old manually settable pulse generator

This is archival information only.  This is no longer used.
  • If the EDTM pulse generator located in rack CH03B06 is reset for any reason, the settings must be restored to the following parameters as outlined in this logbook entry

EDTM pulser settings.jpg

F250 Pedestal Updates

Overview

This How-To discusses the necessary steps required in order to take a pedestal run utilizing the single arm DAQ's. Furthermore, the steps required in-order to determine the appropriate FADC thresholds are discussed.

NOTE: This is for EXPERTS ONLY.  If this is done poorly and not caught the FADCs will not work correctly and the data maybe garbage!

Taking Pedestal Runs

  • Ensure that both single arm DAQ's are configured with the "fadcnothr" flag, which is activated via. of the pre-scale GUI, and the configuration has been saved
    • The FADC's can either be in mode 9 or 10
  • With both arms take at least 1000 cosmic or EDTM triggers and note the run numbers

Determining the FADC thresholds

  • The following example will illustrate the necessary steps required to determine the FADC thresholds for SHMS ROC 2
  • On the SHMS side, this procedure will have to be repeated for both ROC's 2 and 4
  • On the HMS side, this procedure only has to be executed for ROC 1
  • With the pedestal runs complete, open a terminal and SSH into cdaql1
ssh cdaq@cdaql1
  • Setup the hallc-online analyzer
go_analysis
  • Descend into the hallc-online hcana/podd directory
cd ../hcana/podd/hana_decode/apps
  • Exectue the tstfadc_main.C script
./tstfadc
  • Input the appropriate configurations: raw data file location, run number, number of events, spectrometer name, crate number
Enter the location of the raw CODA file (raw, raw.copiedtotape): raw
Enter a Run Number (-1 to exit): 1537 
Enter Number of Events to Analyze (-1 = All): -1
All Events in Run 1537 Will be Analayzed
Enter the Spectrometer Name (hms or shms): shms
Enter the Crate Number to be Analyzed: 2
  • If successful, the for file "fadc_data.root" will have been produced
  • In order to produce the threshold text file "thresholds.dat" one needs to execute the calc_thresh_hallc.C ROOT script
root -l ./calc_thresh_hallc.cxx
  • Input the appropriate configurations: spectrometer name, crate number, FADC mode, threshold in channels
Enter the Spectrometer Name (hms or shms): shms
Enter the Crate Number to be Analyzed: 2
Enter The Mode of The F250's: 9
Enter The Number of Channels Above Baseline the Threshold Should Be [40 (10 mV) recommended]: 40
  • A ROOT file "fadc_ped_data.root" will have been created which contains the final fits and determines the baseline values and thus the threshold values which are written to the text file "thresholds.dat"
  • Open the "thresholds.dat" file and have a sanity check the thresholds
    • Most FADC channels should have thresholds set to 400-700 channels
    • For channels set to 4095, these channels are empty
    • For ROC2 slot 14 channels 12-15 are set to 1 by default since they are fast raster FADC channels
    • For ROC4 slot 18 channels 12-15 are set to 1 by default since they are fast raster FADC channels
  • In the event that a few thresholds are set to unrealistic values, simply open the "fadc_ped_data.root" file and perform the fit manually if possible
    • If a manual fit is not possible, then a new pedestal run with more statistics is recommended

Loading the FADC Thresholds

  • In order to have the CRL's load the appropriate FADC threshold files the newly created "thresholds.dat" file must be placed in the correct location
  • SSH into cdaql3 as coda
ssh coda@cdaql3
  • Descend into the respective thresholds directory for the appropriate ROC
cd coda/config_files/SHMS/ROC02/thresholds
  • Copy the newly created "thresholds.dat" into the current working directory and tag the "thresholds.dat" file with the current date
cp hana_decode/thresholds.dat thresholds_DDMONYYYY.dat
  • Ascend one directory and symbolically link the newly created thresholds file as "thresholds.dat"
cd ../
ln -fs thresholds/thresholds_DDMONYYYY.dat thresholds.dat