Difference between revisions of "Hall C DAQ"
(→SHMS) |
(→NPS) |
||
(138 intermediate revisions by 11 users not shown) | |||
Line 1: | Line 1: | ||
− | = How to | + | = How to kill/reset/restart the DAQ, or, "The DAQ is dead, now what..." = |
− | The | + | |
− | + | ||
− | + | ||
− | ==== | + | === How to connect to VNC session === |
− | + | ||
− | + | ssh into cdaq@cdaql1 and from there type: | |
− | + | go_nps_daq or go_coin_daq (they are the same). | |
+ | However, you most likely already have those screens on one of the hcdesk machines. | ||
− | |||
=== If you are connected to the appropriate DAQ vncsession === | === If you are connected to the appropriate DAQ vncsession === | ||
− | If you are connected to the appropriate DAQ vncsession | + | If you are connected to the appropriate DAQ vncsession and |
− | CODA has hung | + | CODA has hung or gotten into a funny state, then bring a terminal window |
− | + | to the foreground and type the appropriate kill and start script, see below. | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | % killCoin for coincidence DAQ. (kcoda for NPS-standalone) | |
+ | Wait for a bit (10 sec). Then restart coda by typing this in the same terminal: | ||
− | + | % startCoin for coincidence DAQ. (startCoda for NPS-standalone) | |
− | + | A note about the "platform", which is one of the CODA components that runs on the workstation | |
− | + | like cdaql4. It should be running using "systemd". But things can go wrong here, so see the | |
+ | Expert link https://hallcweb.jlab.org/wiki/index.php/Hall_C_DAQ#Expert_notes | ||
− | ==== | + | You will then have to complete the 'standard' CODA steps: |
+ | # Choose 'Platform:Connect' from the CODA Run Control GUI menu bar. [[(Ask the RC which config to use, but the one that comes up is the last config that successfully ran.)]] | ||
+ | # Click on the 'Tools' icon. | ||
+ | # Then you should be able to start a run as usual. | ||
+ | |||
+ | If you the components can't connect, it may be that the platform has a problem. | ||
+ | |||
+ | === If the kill / start process doesn't work === | ||
+ | |||
+ | If the system does not restore using the kcodaCoinc/startCoinc procedure, then you may need to reboot the ROCs | ||
+ | following this process | ||
+ | |||
+ | [[Hall_C_DAQ#If_you_need_to_reboot_a_ROC]]. | ||
+ | |||
+ | === If scalers don't count in-between runs === | ||
+ | |||
+ | First, decide if you care. One reason to care is if you need to adjust prescales. Scalers do count during a run, | ||
+ | or at least I've not seen that fail. Scalers do not count during prestart transition, but that is ok. | ||
+ | |||
+ | If you want to restart scalers (actually the server) wait 2+ minutes after the run and execute the following script, | ||
+ | answering "n" (without the quotes) to confirm that you're not doing a run. (We don't want to kill the server during a run.) Any answer differing from a single lower-case "n" is interpreted as "yes" and the script aborts. | ||
+ | |||
+ | [coda@cdaql4 ~]$ /home/coda/bob/checkServer/restartScalerServer | ||
+ | |||
+ | === If the CODA vnc session disappeared and 'go_coin_daq' doesn't work === | ||
+ | |||
+ | NOTE: Ask a DAQ expert and/or the RC '''before''' you do this. It really should never be needed. | ||
+ | |||
+ | If everything disappeared and 'go_coin_daq' doesn't work then you can bring things up from scratch outside the VNC session. | ||
+ | (It usually means the vncserver didn't start on cdaql4, so you could login to coda@cdaql4 and type /home/coda/coda/scripts/startVnc, | ||
+ | but that's supposed to happen in systemd.) | ||
+ | |||
+ | ==== Further hints on resets ==== | ||
+ | |||
+ | If CODA hangs up | ||
+ | |||
+ | <pre> | ||
+ | 1. Sometimes a simple reset in rcgui fixes it and is the fastest way to restart. It's the backward-arrow in the rcgui (if you let the mouse hover over the buttons it tells you the name of the button). | ||
+ | 2. But if there's a complaint about a VME crate, you may need to power cycle that crate. | ||
+ | 3. Often, even if only one crate complains, one must restart CODA and power-cycle all 8 crates. This means on coda@cdaql4 : killCoinc ; startCoinc ; and | ||
+ | reboot all the crates (see "reboot a ROC" in this page). | ||
+ | </pre> | ||
+ | |||
+ | ==== COIN Mode ==== | ||
+ | This should be our production mode for the experiment. | ||
% ssh coda@cdaql4 | % ssh coda@cdaql4 | ||
In that terminal, run: | In that terminal, run: | ||
− | % | + | % startCoin |
− | ==== | + | Coda configuration in COIN mode |
− | % ssh coda@ | + | coin_sparse : data taking with sparsification on NPS on ( default configuration to use ) |
+ | coin : data taking without sparsification on NPS | ||
+ | coin_vld : LED configuration - make sure to set ps2 to 0 | ||
+ | coin_cosmics : cosmics configuration - make sure to set ps2 to 0 and turn on the cosmics paddles | ||
+ | |||
+ | ==== NPS standalone ==== | ||
+ | (probably not used anymore, see COIN mode above) | ||
+ | % ssh coda@cdaql4 | ||
In that terminal, run: | In that terminal, run: | ||
− | % | + | % startCoda |
− | + | ==== If the Event Builder (PEB) dying or not fully ending ==== | |
+ | |||
+ | There is a chance that when the run ends, the event builder (PEB) will not fully end. It goes to the state "ending" and the DAQ is frozen. The solution is simple: find the event builder (usually a light-blue xterm with the title PEB1), and ctrl-C to exit the event builder, and wait. The PEB will restart on its own and you can start a new run. Sorry, this is a bug in CODA, and the chance of this happening appears to increase if we run with blockLevel>1 and especially if we use USER-end events. | ||
+ | |||
+ | Another possibility is that the event builder may die. This often means there is a memory issue on one of the ROCs. It is simplest to follow the [[Hall_C_DAQ#If_you_need_to_reboot_a_ROC | Reboot ROCs]] procedure to resolve this. | ||
+ | |||
+ | = How to bring up the DAQ screens on a workstation = | ||
+ | The DAQs run inside their own VNC sessions. They can be brought up on any | ||
+ | of the Hall C Counting House machines by running these commands logged in | ||
+ | as the 'cdaq' user: | ||
+ | |||
+ | go_coin_daq ... coincidence configuration. NPS + HMS crates. | ||
+ | |||
+ | go_nps_daq ... NPS standalone. | ||
+ | |||
+ | HMS standalone ? Deprecated, but see https://hallcweb.jlab.org/wiki/index.php/Hall_C_DAQ#Expert_notes | ||
+ | |||
+ | Generally, a crate can only operate with one running DAQ. | ||
= If you need to reboot a ROC = | = If you need to reboot a ROC = | ||
− | + | If a ROC gets in a really bad state, its crate may need to be power cycled using the procedure and links below. Note that often, even if only one crate complains, one must restart CODA and power-cycle all 8 crates. This means on coda@cdaql4 : killCoinc ; startCoinc in addition to power-cycling. | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | It pays to be a little patient with this process. Stagger turning the crate power back on by 10--20 seconds. | |
− | '''' | + | For example, turn on hccrate01, wait 10s, turn on hccrate02, wait 10s, etc... |
+ | |||
+ | '''NOTE''': Open a browser window in the CODA VNC session (usually there is one already running in the left-most workspace). | ||
+ | All of the crate reset links are there along with stored login credentials. | ||
+ | |||
+ | ==== COIN mode ==== | ||
+ | Coincidence (COIN) mode uses ''both'' the '''NPS''' and '''HMS''' ROCs, so you will need to reboot '''all''' of them using the links in the two sub-sections below. | ||
+ | |||
+ | After you have power cycled all the ROCs, wait for 2--3 minutes then run '''kcodaCoin'''; and '''startCoin'''. | ||
* ''Make an hclog when you do this!'' | * ''Make an hclog when you do this!'' | ||
+ | |||
+ | kcodaCoinc is synonymous with killCoinc. | ||
+ | A note: SHMS is not used during the NPS experiments. | ||
==== HMS ==== | ==== HMS ==== | ||
− | The HMS has | + | The HMS has three ROCs. Follow these links to power-cycle: |
* [http://hccrate01 ROC01] (Counting House Electronics Room) | * [http://hccrate01 ROC01] (Counting House Electronics Room) | ||
− | * [http://hccrate03 ROC03] ( | + | * [http://hccrate03 ROC03] (HMS Electronics Hut -- the wire chambers) |
− | * ROC05 (Counting House Electronics Room -- | + | * [http://hccrate05 ROC05] (Counting House Electronics Room) --- On APC switch. See note below. |
− | The | + | ==== NPS ==== |
− | ' | + | The NPS has five ROCs and five VTPs. The VTPs get power cycled when you cycle the ROCs. |
− | the Electronics Room | + | * [http://nps-crate1 npsvme1 and npsvtp1] (NPS crate 1) |
+ | * [http://nps-crate2 npsvme2 and npsvtp2] (NPS crate 2) | ||
+ | * [http://nps-crate3 npsvme3 and npsvtp3] (NPS crate 3) | ||
+ | * [http://nps-crate4 npsvme4 and npsvtp4] (NPS crate 4) | ||
+ | * [http://nps-crate5 npsvme5 and npsvtp5] (NPS crate 5) | ||
+ | |||
+ | Reboot them by clicking on the link and cycling the power. Use the "Main Power" button, not the VME SYSRESET button. '''Wait for 10s after clicking the 'power' button ''before'' clicking it again'''. The page will automatically refresh when it is ready. | ||
+ | |||
+ | After you have power cycled all the ROCs, wait for 2--3 minutes then run the appropriate script (killCoin / startCoin). | ||
+ | |||
+ | '''ROC5/hccrate05''' is on an APC switch. Pick the 'Outlet Status' Tab; click on ''On'' to the left of ''hccrate05'' on Outlet 1. Set the ''Control Action'' to ''Reboot Delayed'', click the checkbox to the left of ''hccrate05'', then click the ''Next>>'' button and click ''Apply'' on the next screen. ''If'' you know where hccrate05 is in the Counting House Electronics Room, you can power cycle it with the switch on the front panel. Turn it off, wait 5 seconds, and turn it | ||
back on. | back on. | ||
− | |||
* ''Make an hclog when you do this!'' | * ''Make an hclog when you do this!'' | ||
+ | |||
+ | ==== Rebooting the DAQ computers ==== | ||
+ | In rare instances, we have to reboot the cdaql5 (HMS) and/or the cdaql4 (coinc or NPS) computers. This is best done under the guidance of the DAQ expert, but the link to the general procedure is documented here: [https://logbooks.jlab.org/entry/4123883 https://logbooks.jlab.org/entry/4123883] | ||
+ | |||
+ | = RCGUI platform cannot connect = | ||
+ | |||
+ | go to expert notes down in this page, and look at how to restart platform instructions | ||
+ | |||
+ | = Live Time Display = | ||
+ | In should be visible on rcgui. There is tab in the lower right area of the GUI. | ||
+ | |||
+ | = Setting Prescales = | ||
+ | [[Image:Prescales.png|thumb]] | ||
+ | Run the prescale GUI using one of these commands from a terminal in the coincidence DAQ session ['''make sure you are using coda@cdaql[1-6]''']. '''Do this from a terminal in the appropriate CODA VNC session.''' | ||
+ | |||
+ | % go_prescale_COIN | ||
+ | |||
+ | <pre> | ||
+ | Sometimes go_prescale_COIN returns an error message below. Sadly, this | ||
+ | might happen several times in a row. Keep trying to start it. Eventually | ||
+ | it will start and it will work. The long-term solution might be to rewrite | ||
+ | this GUI, since the original author / maintainer is long gone. | ||
+ | |||
+ | Error in startup script: can't read "config ... (etc)" | ||
+ | </pre> | ||
+ | |||
+ | == How to pick a prescale value == | ||
+ | The relationship between the prescale setting and the actual prescale value is somewhat complicated. You can click on the 'Help' button for details. | ||
+ | |||
+ | * The best way to do it is to type in the prescaled rate you ''want'' into the '''Target Rate''' field and hit return. The '''Prescaled Rate''' field will update to show you the rate the DAQ will be accepting. | ||
+ | ** You can try adding or subtracting '1' to the 'Value' field if you want to try to tweak the '''Prescaled Rate''' a bit. | ||
+ | |||
+ | NOTE: ''You MUST hit '''SAVE''' *before* starting the run!'' | ||
+ | |||
+ | Details of prescale are in coda-flags.dat attached to each Start-of-Run logbook entry | ||
+ | The new TI has 4 bits to prescale the individual inputs. | ||
+ | ; Set psN=-1 to DISABLE that trigger. | ||
+ | ; Set psN=0 for NO PRESCALING (ie. accept all) | ||
+ | ; The way the input rates are prescaled follows: | ||
+ | ; input-rate/(2^{val - 1} + 1) | ||
+ | ; So, for 10,000 Hz on the input: | ||
+ | ; val scale_factor prescaled rate example | ||
+ | ; 0 -> 1 100kHz ->100000 Hz | ||
+ | ; 1 -> 2 100kHz -> 50000 Hz | ||
+ | ; 2 -> 3 100kHz -> 33333 Hz | ||
+ | ; 3 -> 5 100kHz -> 20000 Hz | ||
+ | ; 4 -> 9 100kHz -> 11111 Hz | ||
+ | ; 5 -> 17 100kHz -> 5882 Hz | ||
+ | ; 6 -> 33 100kHz -> 3030 Hz | ||
+ | ; 16 -> 32769 100kHz -> 3 Hz | ||
+ | |||
+ | == If the scaler rates in the prescale GUI are not updating == | ||
+ | |||
+ | This can happen after CODA gets in a funny state, for example. | ||
+ | |||
+ | * Quit the prescale GUI (Exit button) | ||
+ | * Run the GUI again (as above). | ||
+ | |||
+ | = Expert notes = | ||
+ | |||
+ | Most shifter workers may not need these notes. But it's a reminder to the experts. | ||
+ | |||
+ | How to adjust blockLevel and bufferLevel. | ||
+ | The bufferLevel (normally 10) can still be adjusted with the prescale gui (go_prescale_COIN), which writes to a "flags" file. | ||
+ | The program running on hcvme01 parses this flag file and programs the DAQ. | ||
+ | New to Fall 2023, we can also adjust the blockLevel, but only by editing the flags file directly. An expert can do this. | ||
+ | It's not possible to adjust the blockLevel in the prescale gui. | ||
+ | |||
+ | A note about the vncserver. | ||
+ | It may happen that vncserver didn't start on cdaql4. I think I fixed this, but it's easy to check. Login to coda@cdaql4 and type | ||
+ | "ps axw | grep vnc". If it's not running, type /home/coda/coda/scripts/startVnc. | ||
+ | |||
+ | What if cdaql4 goes bad ? | ||
+ | We should still be able to run on cdaql6 or another cdaq. Just ensure that the environment is valid (see below). There could be | ||
+ | some remnant in the log-in scripts or in coda scripts which relies on cdaql4; ideally not. | ||
+ | <pre> | ||
+ | [coda@cdaql4 ~]$ echo $EXPID | ||
+ | NPS | ||
+ | |||
+ | [coda@cdaql4 ~]$ echo $SESSION | ||
+ | NPS | ||
+ | </pre> | ||
+ | and these are currently "HMS" on cdaql5, which is sometimes used for HMS standalone DAQ. (actually this is deprecated). | ||
+ | |||
+ | <font color="blue">"I'd like to run HMS Standalone DAQ".</font> | ||
+ | |||
+ | You are encouraged to run with the "coinc" configuration whenever you need the HMS crates. It has all the crates and the timing is fine. Still, there could be some reasons to force you to use the HMS Standalone DAQ, which runs on coda@cdaql5 and has EXPID and SESSION set to be HMS. <font color="blue">There are two awkward steps to switching : see https://logbooks.jlab.org/entry/4168506. </font> If there seems to be a big demand for this DAQ, we should try to automate this switchover, so that expert intervention is not needed. However, the expert can do the job in 10 minutes. | ||
+ | |||
+ | Once you do start the HMS-standalone DAQ use | ||
+ | <pre> | ||
+ | HMS-standalone DAQ | ||
+ | Before doing the stuff below, check if HMS-standalone DAQ is already running. | ||
+ | From cdaq@cdaql1 type "go_hms_daq" to open the VNC session for the HMS. If it's not running then | ||
+ | on coda@cdaql5 | ||
+ | killHMS -- to kill processes | ||
+ | startHMS -- to start CODA. | ||
+ | </pre> | ||
+ | |||
+ | Some things to know about the configs: | ||
+ | |||
+ | The "Coinc" and "NPS"-standalone both run on cdaql4 and switching between them is just a matter of picking the config in rcgui. There has been a proliferation of configs. Which config are we using ? Ask the RC to be certain. | ||
+ | The "HMS"-standalone is compatible with running "NPS"=standalone, which would be a motivation for using it. | ||
+ | |||
+ | <font color="magenta">PLATFORM ---- PLATFORM ---- PLATFORM ---- PLATFORM</font> | ||
+ | |||
+ | A note about the <font color="magenta">"platform"</font>. The platform is one of the CODA components that runs on the workstation like cdaql4. It is not killed with "killCoinc" because it's been observed that this causes more trouble than it's worth -- maybe because it starts slowly. However, sometimes it is necessary to kill it, especially after many resets it seems to be necessary to kill and restart the platform. To kill it, look in the process list (ps -awx | grep -ai platform) then "kill -9 PID" for the process ID's (PID) corresponding to the platform process (probably more than 1). You don't have to restart the platform by hand anymore. Instead, the platform should be restarted by the systemd. After killing coda, wait for 10 seconds, check platform status by doing: systemctl status platform. You will hopefully see " Active: active (running)". You could also check the process list, as mentioned above. If the platform is active, start coda. If it's not and you don't know the root password, try this "killall coda_platform.sh", and check platform status to see if it's active. Also, note, the platform should run on the same computer (cdaql4 normally) where you are running the rest of CODA, or else the run-start GUI will not pop up. Also, the platform should not run on any other computer, at least not for this $SESSION. In theory the platform would work elsewhere, but it makes in harder to find it for shift workers (because they may need to kill it), plus the issue of the start-run pop-up. | ||
+ | |||
+ | <pre> | ||
+ | Ole Hansen enabled sudo access for the cdaq and coda accounts on cdaql4 for sysemctl (start|stop|restart) platform.service. So you can do this yourself. If and when necessary, e.g. if it's hung up, terminate the running platform process, along these lines: | ||
+ | |||
+ | [coda@cdaql4 ~]# ps auxwwf | grep platform | ||
+ | coda 68129 0.0 0.0 119400 1352 pts/16 S+ Aug21 0:00 | \_ /bin/csh -f /home/coda/coda/3.10_devel/Linux-x86_64/bin/platform | ||
+ | coda 68131 3.2 2.7 8355612 435436 pts/16 Sl+ Aug21 68:44 | \_ /home/coda/coda/jdk1.8.0_152/bin/java -Xms200m -Xmx2048m -Djava.net.preferIPv4Stack=true org.jlab.coda.afecs.platform.APlatform | ||
+ | |||
+ | [coda@cdaql4 ~]# kill 68129 68131 | ||
+ | |||
+ | [coda@cdaql4 ~]# sudo systemctl start platform.service | ||
+ | |||
+ | [coda@cdaql4 ~]# systemctl status platform.service | ||
+ | </pre> | ||
+ | |||
+ | If you don't know the root password, try the following to restart platform: | ||
+ | <pre> | ||
+ | killall coda_platform.sh | ||
+ | </pre> | ||
+ | |||
+ | <font color="magenta">Scripts.</font> | ||
+ | |||
+ | The hall C philosophy is to call one script at the start of run and call another script at the end of run. These scripts, in turn, call several other scripts. | ||
+ | CODA experts familiar with jcedit will realize what the following snapshots mean. Caution! only experts should change things | ||
+ | How scripts tied to event builder PEB http://userweb.jlab.org/~rom/hallc/PEB_process.png | ||
+ | Process 1 is start_logruninfo.sh and various versions of that like start_logruninfo_nps_coin.sh http://userweb.jlab.org/~rom/hallc/PEB_process1_start_logruninfo.sh.png | ||
+ | Process 2 is the end-of-run script http://userweb.jlab.org/~rom/hallc/process2_end_of_run-master.sh.png | ||
+ | Look at what's actually in the database, as these things tend to evolve. Experts only. | ||
+ | |||
+ | MSS copying script. All the "action" is on coda@cdaql4. It's a cron job on the coda account, type "crontab -l" to see. The main script we care about for moving data to MSS and deleting from disk is: /home/coda/coda/jmirror_scripts/hallc_jmirror. This is a lot like the setup in hall A. Yes, I did change Brad's scheme to 100% use jmirror. | ||
+ | |||
+ | = EDTM Settings = | ||
+ | {{:Hall_C_EDTM_Pulser}} | ||
+ | |||
+ | = F250 Pedestal Updates = | ||
+ | {{:Hall_C_Pedestal_Runs_and_Determining_FADC_Thresholds}} |
Latest revision as of 18:07, 31 January 2024
Contents
- 1 How to kill/reset/restart the DAQ, or, "The DAQ is dead, now what..."
- 2 How to bring up the DAQ screens on a workstation
- 3 If you need to reboot a ROC
- 4 RCGUI platform cannot connect
- 5 Live Time Display
- 6 Setting Prescales
- 7 Expert notes
- 8 EDTM Settings
- 9 F250 Pedestal Updates
How to kill/reset/restart the DAQ, or, "The DAQ is dead, now what..."
How to connect to VNC session
ssh into cdaq@cdaql1 and from there type: go_nps_daq or go_coin_daq (they are the same).
However, you most likely already have those screens on one of the hcdesk machines.
If you are connected to the appropriate DAQ vncsession
If you are connected to the appropriate DAQ vncsession and CODA has hung or gotten into a funny state, then bring a terminal window to the foreground and type the appropriate kill and start script, see below.
% killCoin for coincidence DAQ. (kcoda for NPS-standalone)
Wait for a bit (10 sec). Then restart coda by typing this in the same terminal:
% startCoin for coincidence DAQ. (startCoda for NPS-standalone)
A note about the "platform", which is one of the CODA components that runs on the workstation like cdaql4. It should be running using "systemd". But things can go wrong here, so see the Expert link https://hallcweb.jlab.org/wiki/index.php/Hall_C_DAQ#Expert_notes
You will then have to complete the 'standard' CODA steps:
- Choose 'Platform:Connect' from the CODA Run Control GUI menu bar. (Ask the RC which config to use, but the one that comes up is the last config that successfully ran.)
- Click on the 'Tools' icon.
- Then you should be able to start a run as usual.
If you the components can't connect, it may be that the platform has a problem.
If the kill / start process doesn't work
If the system does not restore using the kcodaCoinc/startCoinc procedure, then you may need to reboot the ROCs following this process
Hall_C_DAQ#If_you_need_to_reboot_a_ROC.
If scalers don't count in-between runs
First, decide if you care. One reason to care is if you need to adjust prescales. Scalers do count during a run, or at least I've not seen that fail. Scalers do not count during prestart transition, but that is ok.
If you want to restart scalers (actually the server) wait 2+ minutes after the run and execute the following script, answering "n" (without the quotes) to confirm that you're not doing a run. (We don't want to kill the server during a run.) Any answer differing from a single lower-case "n" is interpreted as "yes" and the script aborts.
[coda@cdaql4 ~]$ /home/coda/bob/checkServer/restartScalerServer
If the CODA vnc session disappeared and 'go_coin_daq' doesn't work
NOTE: Ask a DAQ expert and/or the RC before you do this. It really should never be needed.
If everything disappeared and 'go_coin_daq' doesn't work then you can bring things up from scratch outside the VNC session. (It usually means the vncserver didn't start on cdaql4, so you could login to coda@cdaql4 and type /home/coda/coda/scripts/startVnc, but that's supposed to happen in systemd.)
Further hints on resets
If CODA hangs up
1. Sometimes a simple reset in rcgui fixes it and is the fastest way to restart. It's the backward-arrow in the rcgui (if you let the mouse hover over the buttons it tells you the name of the button). 2. But if there's a complaint about a VME crate, you may need to power cycle that crate. 3. Often, even if only one crate complains, one must restart CODA and power-cycle all 8 crates. This means on coda@cdaql4 : killCoinc ; startCoinc ; and reboot all the crates (see "reboot a ROC" in this page).
COIN Mode
This should be our production mode for the experiment.
% ssh coda@cdaql4
In that terminal, run:
% startCoin
Coda configuration in COIN mode
coin_sparse : data taking with sparsification on NPS on ( default configuration to use ) coin : data taking without sparsification on NPS coin_vld : LED configuration - make sure to set ps2 to 0 coin_cosmics : cosmics configuration - make sure to set ps2 to 0 and turn on the cosmics paddles
NPS standalone
(probably not used anymore, see COIN mode above)
% ssh coda@cdaql4
In that terminal, run:
% startCoda
If the Event Builder (PEB) dying or not fully ending
There is a chance that when the run ends, the event builder (PEB) will not fully end. It goes to the state "ending" and the DAQ is frozen. The solution is simple: find the event builder (usually a light-blue xterm with the title PEB1), and ctrl-C to exit the event builder, and wait. The PEB will restart on its own and you can start a new run. Sorry, this is a bug in CODA, and the chance of this happening appears to increase if we run with blockLevel>1 and especially if we use USER-end events.
Another possibility is that the event builder may die. This often means there is a memory issue on one of the ROCs. It is simplest to follow the Reboot ROCs procedure to resolve this.
How to bring up the DAQ screens on a workstation
The DAQs run inside their own VNC sessions. They can be brought up on any of the Hall C Counting House machines by running these commands logged in as the 'cdaq' user:
go_coin_daq ... coincidence configuration. NPS + HMS crates.
go_nps_daq ... NPS standalone.
HMS standalone ? Deprecated, but see https://hallcweb.jlab.org/wiki/index.php/Hall_C_DAQ#Expert_notes
Generally, a crate can only operate with one running DAQ.
If you need to reboot a ROC
If a ROC gets in a really bad state, its crate may need to be power cycled using the procedure and links below. Note that often, even if only one crate complains, one must restart CODA and power-cycle all 8 crates. This means on coda@cdaql4 : killCoinc ; startCoinc in addition to power-cycling.
It pays to be a little patient with this process. Stagger turning the crate power back on by 10--20 seconds. For example, turn on hccrate01, wait 10s, turn on hccrate02, wait 10s, etc...
NOTE: Open a browser window in the CODA VNC session (usually there is one already running in the left-most workspace). All of the crate reset links are there along with stored login credentials.
COIN mode
Coincidence (COIN) mode uses both the NPS and HMS ROCs, so you will need to reboot all of them using the links in the two sub-sections below.
After you have power cycled all the ROCs, wait for 2--3 minutes then run kcodaCoin; and startCoin.
- Make an hclog when you do this!
kcodaCoinc is synonymous with killCoinc. A note: SHMS is not used during the NPS experiments.
HMS
The HMS has three ROCs. Follow these links to power-cycle:
- ROC01 (Counting House Electronics Room)
- ROC03 (HMS Electronics Hut -- the wire chambers)
- ROC05 (Counting House Electronics Room) --- On APC switch. See note below.
NPS
The NPS has five ROCs and five VTPs. The VTPs get power cycled when you cycle the ROCs.
- npsvme1 and npsvtp1 (NPS crate 1)
- npsvme2 and npsvtp2 (NPS crate 2)
- npsvme3 and npsvtp3 (NPS crate 3)
- npsvme4 and npsvtp4 (NPS crate 4)
- npsvme5 and npsvtp5 (NPS crate 5)
Reboot them by clicking on the link and cycling the power. Use the "Main Power" button, not the VME SYSRESET button. Wait for 10s after clicking the 'power' button before clicking it again. The page will automatically refresh when it is ready.
After you have power cycled all the ROCs, wait for 2--3 minutes then run the appropriate script (killCoin / startCoin).
ROC5/hccrate05 is on an APC switch. Pick the 'Outlet Status' Tab; click on On to the left of hccrate05 on Outlet 1. Set the Control Action to Reboot Delayed, click the checkbox to the left of hccrate05, then click the Next>> button and click Apply on the next screen. If you know where hccrate05 is in the Counting House Electronics Room, you can power cycle it with the switch on the front panel. Turn it off, wait 5 seconds, and turn it back on.
- Make an hclog when you do this!
Rebooting the DAQ computers
In rare instances, we have to reboot the cdaql5 (HMS) and/or the cdaql4 (coinc or NPS) computers. This is best done under the guidance of the DAQ expert, but the link to the general procedure is documented here: https://logbooks.jlab.org/entry/4123883
RCGUI platform cannot connect
go to expert notes down in this page, and look at how to restart platform instructions
Live Time Display
In should be visible on rcgui. There is tab in the lower right area of the GUI.
Setting Prescales
Run the prescale GUI using one of these commands from a terminal in the coincidence DAQ session [make sure you are using coda@cdaql[1-6]]. Do this from a terminal in the appropriate CODA VNC session.
% go_prescale_COIN
Sometimes go_prescale_COIN returns an error message below. Sadly, this might happen several times in a row. Keep trying to start it. Eventually it will start and it will work. The long-term solution might be to rewrite this GUI, since the original author / maintainer is long gone. Error in startup script: can't read "config ... (etc)"
How to pick a prescale value
The relationship between the prescale setting and the actual prescale value is somewhat complicated. You can click on the 'Help' button for details.
- The best way to do it is to type in the prescaled rate you want into the Target Rate field and hit return. The Prescaled Rate field will update to show you the rate the DAQ will be accepting.
- You can try adding or subtracting '1' to the 'Value' field if you want to try to tweak the Prescaled Rate a bit.
NOTE: You MUST hit SAVE *before* starting the run!
Details of prescale are in coda-flags.dat attached to each Start-of-Run logbook entry
The new TI has 4 bits to prescale the individual inputs.
- Set psN=-1 to DISABLE that trigger.
- Set psN=0 for NO PRESCALING (ie. accept all)
- The way the input rates are prescaled follows
- input-rate/(2^{val - 1} + 1)
- So, for 10,000 Hz on the input
- val scale_factor prescaled rate example
- 0 -> 1 100kHz ->100000 Hz
- 1 -> 2 100kHz -> 50000 Hz
- 2 -> 3 100kHz -> 33333 Hz
- 3 -> 5 100kHz -> 20000 Hz
- 4 -> 9 100kHz -> 11111 Hz
- 5 -> 17 100kHz -> 5882 Hz
- 6 -> 33 100kHz -> 3030 Hz
- 16 -> 32769 100kHz -> 3 Hz
If the scaler rates in the prescale GUI are not updating
This can happen after CODA gets in a funny state, for example.
- Quit the prescale GUI (Exit button)
- Run the GUI again (as above).
Expert notes
Most shifter workers may not need these notes. But it's a reminder to the experts.
How to adjust blockLevel and bufferLevel. The bufferLevel (normally 10) can still be adjusted with the prescale gui (go_prescale_COIN), which writes to a "flags" file. The program running on hcvme01 parses this flag file and programs the DAQ. New to Fall 2023, we can also adjust the blockLevel, but only by editing the flags file directly. An expert can do this. It's not possible to adjust the blockLevel in the prescale gui.
A note about the vncserver. It may happen that vncserver didn't start on cdaql4. I think I fixed this, but it's easy to check. Login to coda@cdaql4 and type "ps axw | grep vnc". If it's not running, type /home/coda/coda/scripts/startVnc.
What if cdaql4 goes bad ? We should still be able to run on cdaql6 or another cdaq. Just ensure that the environment is valid (see below). There could be some remnant in the log-in scripts or in coda scripts which relies on cdaql4; ideally not.
[coda@cdaql4 ~]$ echo $EXPID NPS [coda@cdaql4 ~]$ echo $SESSION NPS
and these are currently "HMS" on cdaql5, which is sometimes used for HMS standalone DAQ. (actually this is deprecated).
"I'd like to run HMS Standalone DAQ".
You are encouraged to run with the "coinc" configuration whenever you need the HMS crates. It has all the crates and the timing is fine. Still, there could be some reasons to force you to use the HMS Standalone DAQ, which runs on coda@cdaql5 and has EXPID and SESSION set to be HMS. There are two awkward steps to switching : see https://logbooks.jlab.org/entry/4168506. If there seems to be a big demand for this DAQ, we should try to automate this switchover, so that expert intervention is not needed. However, the expert can do the job in 10 minutes.
Once you do start the HMS-standalone DAQ use
HMS-standalone DAQ Before doing the stuff below, check if HMS-standalone DAQ is already running. From cdaq@cdaql1 type "go_hms_daq" to open the VNC session for the HMS. If it's not running then on coda@cdaql5 killHMS -- to kill processes startHMS -- to start CODA.
Some things to know about the configs:
The "Coinc" and "NPS"-standalone both run on cdaql4 and switching between them is just a matter of picking the config in rcgui. There has been a proliferation of configs. Which config are we using ? Ask the RC to be certain. The "HMS"-standalone is compatible with running "NPS"=standalone, which would be a motivation for using it.
PLATFORM ---- PLATFORM ---- PLATFORM ---- PLATFORM
A note about the "platform". The platform is one of the CODA components that runs on the workstation like cdaql4. It is not killed with "killCoinc" because it's been observed that this causes more trouble than it's worth -- maybe because it starts slowly. However, sometimes it is necessary to kill it, especially after many resets it seems to be necessary to kill and restart the platform. To kill it, look in the process list (ps -awx | grep -ai platform) then "kill -9 PID" for the process ID's (PID) corresponding to the platform process (probably more than 1). You don't have to restart the platform by hand anymore. Instead, the platform should be restarted by the systemd. After killing coda, wait for 10 seconds, check platform status by doing: systemctl status platform. You will hopefully see " Active: active (running)". You could also check the process list, as mentioned above. If the platform is active, start coda. If it's not and you don't know the root password, try this "killall coda_platform.sh", and check platform status to see if it's active. Also, note, the platform should run on the same computer (cdaql4 normally) where you are running the rest of CODA, or else the run-start GUI will not pop up. Also, the platform should not run on any other computer, at least not for this $SESSION. In theory the platform would work elsewhere, but it makes in harder to find it for shift workers (because they may need to kill it), plus the issue of the start-run pop-up.
Ole Hansen enabled sudo access for the cdaq and coda accounts on cdaql4 for sysemctl (start|stop|restart) platform.service. So you can do this yourself. If and when necessary, e.g. if it's hung up, terminate the running platform process, along these lines: [coda@cdaql4 ~]# ps auxwwf | grep platform coda 68129 0.0 0.0 119400 1352 pts/16 S+ Aug21 0:00 | \_ /bin/csh -f /home/coda/coda/3.10_devel/Linux-x86_64/bin/platform coda 68131 3.2 2.7 8355612 435436 pts/16 Sl+ Aug21 68:44 | \_ /home/coda/coda/jdk1.8.0_152/bin/java -Xms200m -Xmx2048m -Djava.net.preferIPv4Stack=true org.jlab.coda.afecs.platform.APlatform [coda@cdaql4 ~]# kill 68129 68131 [coda@cdaql4 ~]# sudo systemctl start platform.service [coda@cdaql4 ~]# systemctl status platform.service
If you don't know the root password, try the following to restart platform:
killall coda_platform.sh
Scripts.
The hall C philosophy is to call one script at the start of run and call another script at the end of run. These scripts, in turn, call several other scripts. CODA experts familiar with jcedit will realize what the following snapshots mean. Caution! only experts should change things How scripts tied to event builder PEB http://userweb.jlab.org/~rom/hallc/PEB_process.png Process 1 is start_logruninfo.sh and various versions of that like start_logruninfo_nps_coin.sh http://userweb.jlab.org/~rom/hallc/PEB_process1_start_logruninfo.sh.png Process 2 is the end-of-run script http://userweb.jlab.org/~rom/hallc/process2_end_of_run-master.sh.png Look at what's actually in the database, as these things tend to evolve. Experts only.
MSS copying script. All the "action" is on coda@cdaql4. It's a cron job on the coda account, type "crontab -l" to see. The main script we care about for moving data to MSS and deleting from disk is: /home/coda/coda/jmirror_scripts/hallc_jmirror. This is a lot like the setup in hall A. Yes, I did change Brad's scheme to 100% use jmirror.
EDTM Settings
Remotely adjustable EDTM
Currently set with a a command-line helper go_edtm
- There is a command-line helper
go_edtm
to set/readback this PV. (the prescale_GUI does not set it at the moment.) - More details are in this log entry: EDTM pulser switched to cdaqpi1 output
- The RPi that controls this is cdaqpi1 which runs an EPICS softIOC.
F250 Pedestal Updates
Overview
This How-To discusses the necessary steps required in order to take a pedestal run utilizing the single arm DAQ's. Furthermore, the steps required in-order to determine the appropriate FADC thresholds are discussed.
NOTE: This is for EXPERTS ONLY. If this is done poorly and not caught the FADCs will not work correctly and the data maybe garbage!
Taking Pedestal Runs
- Ensure that both single arm DAQ's are configured with the "fadcnothr" flag, which is activated via. of the pre-scale GUI, and the configuration has been saved
- The FADC's can either be in mode 9 or 10
- With both arms take at least 1000 cosmic or EDTM triggers and note the run numbers
Determining the FADC thresholds
- The following example will illustrate the necessary steps required to determine the FADC thresholds for SHMS ROC 2
- On the SHMS side, this procedure will have to be repeated for both ROC's 2 and 4
- On the HMS side, this procedure only has to be executed for ROC 1
- With the pedestal runs complete, open a terminal and SSH into cdaql1
ssh cdaq@cdaql1
- Setup the hallc-online analyzer
go_analysis
- Descend into the hallc-online hcana/podd directory
cd ../hcana/podd/hana_decode/apps
- Exectue the tstfadc_main.C script
./tstfadc
- Input the appropriate configurations: raw data file location, run number, number of events, spectrometer name, crate number
Enter the location of the raw CODA file (raw, raw.copiedtotape): raw Enter a Run Number (-1 to exit): 1537 Enter Number of Events to Analyze (-1 = All): -1 All Events in Run 1537 Will be Analayzed Enter the Spectrometer Name (hms or shms): shms Enter the Crate Number to be Analyzed: 2
- If successful, the for file "fadc_data.root" will have been produced
- In order to produce the threshold text file "thresholds.dat" one needs to execute the calc_thresh_hallc.C ROOT script
root -l ./calc_thresh_hallc.cxx
- Input the appropriate configurations: spectrometer name, crate number, FADC mode, threshold in channels
Enter the Spectrometer Name (hms or shms): shms Enter the Crate Number to be Analyzed: 2 Enter The Mode of The F250's: 9 Enter The Number of Channels Above Baseline the Threshold Should Be [40 (10 mV) recommended]: 40
- A ROOT file "fadc_ped_data.root" will have been created which contains the final fits and determines the baseline values and thus the threshold values which are written to the text file "thresholds.dat"
- Open the "thresholds.dat" file and have a sanity check the thresholds
- Most FADC channels should have thresholds set to 400-700 channels
- For channels set to 4095, these channels are empty
- For ROC2 slot 14 channels 12-15 are set to 1 by default since they are fast raster FADC channels
- For ROC4 slot 18 channels 12-15 are set to 1 by default since they are fast raster FADC channels
- In the event that a few thresholds are set to unrealistic values, simply open the "fadc_ped_data.root" file and perform the fit manually if possible
- If a manual fit is not possible, then a new pedestal run with more statistics is recommended
Loading the FADC Thresholds
- These instructions only pertain to the HMS at the moment. For NPS, it is different.
- In order to have the CRL's load the appropriate FADC threshold files the newly created "thresholds.dat" file must be placed in the correct location
- SSH into cdaql1 as coda
ssh coda@cdaql1
- Descend into the respective thresholds directory for the appropriate ROC
cd coda/config_files/SHMS/ROC02/thresholds
- Copy the newly created "thresholds.dat" into the current working directory and tag the "thresholds.dat" file with the current date
cp hana_decode/thresholds.dat thresholds_DDMONYYYY.dat
- Ascend one directory and symbolically link the newly created thresholds file as "thresholds.dat"
cd ../ ln -fs thresholds/thresholds_DDMONYYYY.dat thresholds.dat