Problem Reports & Solutions &
Hot tos & Stuff
12 GeV: Condense H2 when another loop is cold GHe
12 GeV: Restart_ALH_alarm_sound:
12
GeV: Intermittent ALH white-outs
(like every few hours as we had Feb 2019) on all alarms in the alarm handler.
Qweak target: Resetting the horizontal
motion after it has slipped.
Qweak
target: Amp Fault/Stuck lifter:
Qweak target: Pump/Fan Reset
(LH2 recirculation pump):
Qweak target: Turbopump reset
Qweak target: IOC reboot
Qweak target: Target computer (poltarcc) crashes
Qweak target: Cooldown tricks
Qweak target: Encoder Table
(Last updated Feb 10, 2011)
Qweak target: Reload
the alarm limits
Qweak target: Grab the
screens for an hclog
Qweak target: Move to a
different workspace
G0 target: Opening
the JT actually closes it
G0 target: Site-wide
Power Failure
G0 target: typical
warmup
G0 target: typical cooldown , here too
G0 target: target
scan
G0 target: Changing the alarm
limit files
G0 target: Tach problems
G0 target: MedmQ
G0 target: New HPH PS pin
assignments, and problem (and fix)
with power calculation
G0 target: Mystery
pressure drop when going to C or out-of-beam position
G0 target: ITCs hang,
heater goes beserk
G0 traget: Coolant xfer line insulating vacuum
G0 target: Gas target
running
G0 target: House
helium leak
G0 target: xfer line IV
G0 target: No Striptool,
readout errors stuck on
Any tgt: Frozen computer (eg
jeffylab)? Then restart X by typing `control-alt-backspace' ,
then type `startx'.
Sp03 tgt: GRS notes on the cooldown procedure
SP03 tgt: GRS notes on the warmup
procedure
SP03 tgt: GRS notes on the ALH
files & Checklist
SP03 tgt: Can't
operate JT valve
SP03 tgt: Yellow alarm handler
light won't go away.
SP03 tgt: Can't
start alarm handler. Can't start the stripcharts.
SP03 tgt: IOC won't
reboot.
SP03 tgt: Cryostack
lifter does not work.
SP03 tgt: JT PID for parking the
target on autopilot:
SP03 tgt: Shorted high power
heater
SM04 tgt: Archiver stopped.
SM04 tgt: Scattering
chamber vacuum.
SM04 tgt:
Any tgt: Terminal
Bell volume
Any tgt: Automatic
Paging
Any tgt: Flatline Alarm
Any tgt: Beam Current & Raster
Alarms
Any tgt: Turbopump
or vacuum problems
Any tgt: Power
cycle the fans
FAQs:
Condense LH2 when another
loop has cold GHe (for experts only!): Assume
loop1 is cold ~20K GHe and you want to condense H2 in
loop2. Assume loop2 has already been pumped & purged
and is open to the ballast tank. The hall should be in controlled access.
1. Turn the loop2 aux HPH pot to
zero, and turn on the loop2 aux HPH ps using the
covered switch in the counting room. It HAS to be ON in the counting room
BEFORE you turn on the aux HPH in the hall, or else it won`t come up enabled
and you`ll be unable to control it later from the counting room. (Jan 2019
fix).
2. Then open loop1 JT to 12 or 13 or 14 g/s to get the xfer
lines nice and cold. Then crack the ``warm`` loop2 (it`ll be ~ 50 K or so) just
a tiny bit, like 0.5-1%, to start some flow. Keep the return temp below 30 K.
When the pulse of warm gas has passed (could take awhile,
and it could be sudden so watch it carefully), start to open the loop2 JT a bit
more, taking some away from loop1.
3. When you hit the VP curve and
start condensing, go down into the hall. Turn on the loop2 HPH and the loop2
aux HPH ps`s. Stand in front of the fan controllers.
Flip on the loop2 fan power switch down by your ankles. Go back to the counting
room. Turn off the loop2 aux HPH switch. Open the loop 2 fan GUI and turn it on
to 30, then 40, then 50 Hz. That will add heat, which is why we waited until
now to turn it on.
4. Make sure the PID is on, and
continue to condense.
5. Note that to warm up (backout), you HAVE to turn off the fan from the GUI before you turn it off from the
aux HPH panel in the counting room (using the kill switches for the fans) or
you risk damaging the fans.
Restart_ALH_alarm_sound: Type ‟pulseaudio −k″
to restart sound daemon. Test beeping either from alarm
handler setup menu, or typing ‟play /usr/share/sounds/purple/alert.wav″.
ALH
whiteouts: (Solved by Brad S.) These
symptoms have been caused by short (default) connection time-out settings in
the network switches that route date between Hall C and the Accelerator
subnets. The session timeouts on both network switches should be set to 24
hours.
There
are at least two switches in that path, one supports Hall traffic and is
owned/managed by the JLab CNI (Central Network Infrastructure)
group -- contact Bryan Hess and/or Carl Bolitho to check/fix those settings. The second is owned/managed
by the Accelerator group -- contact Brad Cumbia to fix/check the issue on that
side.
Encoder table:
The encoder table for all the target positions is here
(Feb 10, 2011).
Switch workspace on poltarcc:
Use cntrl-alt-rightarrow to
move to the workspace to the right.
Screen grab for the hclog on poltarcc: Use
the printscreen button on the keyboard to grab the
picture of both screens (striptools & main GUI).
If you just want to grab a single screen, press alt-printscreen.
Then save the images, and make an hclog from the
browser. Attach the file(s) you just saved.
Amp fault/stuck lifter: Hit
the big red panic button on the left side of the U-shaped console in the
counting room to power cycle the 6961 controller. Then pull it back out! You
may want to look at the controller with the TV to insure it is back on, and out
of fault condition. Then go "HOME" on the Main Gui
target selection. If you look at the lights above the 6961 controller (with the
TV camera), when not moving the target the 2 middle
lights ("no fault", and "power cycle") should be lit green.
Reset the LH2 recirculation pump/fan: Bring
up the pump GUI from the Main GUI. Hit "reset", that usually does the
trick. If not try "clear" and "Update". If that fails try
entering a new fan frequency (~ 3* what you want in Hz
for the tach) and try stopping and restarting. Look at the TV camera to see the
tach in the hall to see if it is starting. Note that it may take a few seconds
to see it start to move after a successful action. If all else
fails hit the big red pump panic button on the center section of the U-shaped
console in the counting room (and then pull it back out). Then go back and try
to start it the usual ways. For a summary of the instructions, see http://www.jlab.org/~smithg/target/Qweak/IFTHELH2PUMPTRIPS.pdf.
Vacuum Problem: reset turbos
or open turbo gate valve: See target web page. The downstream beamline turbopump controller and CCG controller are in the main
detector hut, beam left side, near the back at ankle height on the FW platform.
You can see it as you enter the MD hut.
The turbopump controllers for the scattering chamber,
surge tank, and (for now) the 15K supply xfer line
are located just behind the standard pivot gas panel in a blue rack about head
height. Walk straight ahead as you enter the hall and
you will run into this gas panel, and the blue rack just behind it is where
these controllers are. The CCG controllers for these are in the array of other
CCG controllers just to the left of the target computer in the target section
of the blue racks in the hall, just to the right as you enter the hall.
If the turbo controller has tripped, you can see it on the Hall C/Qweak vacuum GUI from edmmonticello.
In the hall, on an escorted (quick) access, you will see a fault light on the
controller is lit. Press
"Stop", wait a few minutes for it to spin down, then hit
"Start". The lights will cycle up the diagonal until all are lit. This will take 5 minutes or so. Then they will
start un-lighting until only the bottom load
light is left lit.
Finally, there is a gate valve between each turbopump and the space it pumps on. A common
control is used for the 3 on the target chamber, it is
on the 1st floor of the blue platform, essentially right above the 3 turbopump controllers. Red is closed,
if it is open there are no lights cuz the open light
is burned out. SOmetimes this GV closes, and you may
need to reopen it by hand in the hall. Just to the left of this valve you will
also find the backing pump and blower shared by all 3
of the turbos.
Cooldown
tricks: Open MV28 and MV2, leave MV3 closed. Close MV28, MV2 (and MV3)
after cooldown, and watch out you will get 1 kW extra
on the heater when you do that. Use 5-10 g/s 4K, 10-15 g/s 15K on cold return,
8 g/s while on WR. Close 4K bypass when 4K return temp
at ESR begins to drop. Run pump at 30 Hz early to get some head earlier. Phone
numbers: Errol 7058.
IOC reboot: Press reset
button the the IOC, which is in the electronics room
next to the counting room. You'll see it as you pass
through the star trek doors immediately on your left, at head height. Carefully
poke your finger under the IOC ribbon cable and press the "reset"
button. Use the pot in the counting room to control the auxiliary heater power
supply and supply heat to the heater in the target during the reboot. You have
to switch this on, and off again when you are done. You may wish to switch off
the main heater while you do this. Keep an eye on the target temps
just to the left of this pot. These temps will tell
you what is going on while the IOC is down. They don't
go through a computer to get to those red LED displays.
Target controls computer crash
recovery: If the target controls computer goes down for whatever
reason, slide over to the cdaql6 terminal. Login as poltar@cdaql6 with the
usual un/pw. Then type "cd Desktop/Screeens".
Then type "./tgtgui".
Now you can monitor and control the target from cdaql6 while you take your time
resetting the target controls computer, poltarcc.
Note you can also do this on poltarch, the target
computer in the hall., by ssh'ing
to it from anywhere.
Reload the alarm limits:
Move to Desktop/Screens/ALH. If you need to change an existing alarm limit,
just emacs the text file alarmlim-coldH2.csh
which just consists of a bunch of caputs. Then
get out and go back a level to Desktop/Screens. First type "source
setup" to establish the EPICS addresses you need. Then type "source
./ALH/alarmlim-coldH2.csh". To add new alarms, emacs
the file qweak.alhConfig in the same directory.
Parrot the syntax used for the other alarms. See hclog
227303.
Horizontal Motion reset after
it has slipped:
1. Click "Horiz. Motion." Use this gui to move the target
horizontally by putting step numbers in the "Absolute Position:" box.
2. Have the qwkphytronXpert.adl gui opened. It should look like this.
And the qwkphytronParams.adl
should look like this.
3. Keep moving to negative step numbers until LS- is hit, write down the
"Actual Steps:"
Also Xpot should show about 4.3-4.5
4. Go to the Xpert gui and in the command line PC? put
PC-66628 (no spaces) and hit ENTER. Then put the command PC? and
hit ENTER. The read-back in "Actual Steps:" box should change to
-66628 (this resets the step counter to the calibrated value for the LS-).
5. Click Reset button and then click "Lim SW reset" button. It shoudl clear "Limit switch -" red light.
6. At this stage you can either move it by hand or from the main gui to check it out.
7. Once you moved it away from LS- and still see an Error red
light click Status, if it has a red light for "Limit switch error"
then click "Reset" and "Lim SW reset" again. This
should clear it.
The difference between actual steps at 3. and 66628 divided by 1600 is how much it slipped in mm.
The LH2 Xpot is 48.50 - 48.60
Archiver stuck: Check status at http://jeffylab:4812/ Go to
Groups, Target.
To restart archiver, on jeffylab, ~/Archives, type ./start_archiver
(that stops & starts it). There is also a ./stop_archiver, by the way.
How to print or make the Checklist: cd
$GUI, ./Make_Checklist.bash
To print: rm
~/Checklist, ./Make_Checklist.bash
> ~/Checklist, pdq -Pchchp2
~/Checklist
Can't operate JT valve: Usually because the
step size is zero. One operator had problems entering a step size because they
had forgotten that you have to double-click in the step-size field, enter the
step size, then hit enter all while the mouse is still in that step-size field.
Stuck alarm handler: Sometimes after an
alarm, you can't clear the light in the alarm handler.
Just stop and restart the alarm handler from the main gui.
Can't
start alarm handler and/or stripcharts:
Usually because someone killed the xterm window from
which the main GUI was launched. When this happens, kill the main gui and everything else, bring up
a new xterm, restart the main gui
and everything else. Note that this xterminal has to
stay alive in order to receive diagnostic messages. If it is dismissed, things won't work or won't start like the alarm handler and the stripcharts.
IOC won't reboot:
Ususally because the network is down. In this case,
closely monitor the TV temps on the ITCs and adjust
the auxillary heat load pots by hand to keep the
temperatures near their goal values. Retry the IOC reboot every 5 or 10 minutes until the network is back and the reboot
succeeds. Can also be the optilogic box behind the
furthest left blue racks in the electronics room. Go
thru the startrek doors, turn left, walk to the wall. Go behind the racks. In the middle rack,
about chest high, is some black box with the word optilogic
on it. Pull the power cable to power cycle it and you ought to be back in
business.
Cryostack
lifter doesn't work: Usually this is because
someone has pressed (accidentally) one of the several emergency buttons which
kill the power to the lifter. See for example this hclog
entry.
JT PID: This JT pid was tried out over the 2003 open house. It
adjusts one of the JTs to keep the return temperature at 30K. This way the
targets can all be left safely as gas, not liquid. The
pid parameters are given in hclog 60105.
Shorted HPH: Symptoms of the short we
had in the HPH can be found in hclog entries 58828
, 58857
,
Changing
default alarm limits: In /group/gzero/g0targ/bin/alarmlim-cold-H2.csh,
edit the caputs to reflect the new alarm limits you
want to overide the existing defaults. As
g0targ@gzerol2.acc.jlab.org.
Tach Problems: In the main gui "general" button,
under "tachometer expert". After a power cycle
it may come back in local instead of remote mode. You can reset it from this
GUI.
Or, you can filter out the noise that is there on the
readout sometimes. Note that an IOC reboot can sometimes reduce the noise. But if it persists, the options are:
The smoothing factor is a configurable value between 0 and 1
were 0 means no smoothing and 1 is ultimate smoothing (in fact, the data value
will never change)..default will be .95
After a reboot, sometimes the tach readback is so
noisy even the filtering is useless. Another reboot will solve this problem. Don't ask me why, but this ALWAYS works!
:MedmQ
Starts the main G0 target GUI without all the strip charts and without the
alarm handler. Useful when running/monitoring from home, for example.
New G0 HPH PS: The current reaback is flakey. So new variables ghptrue_ca
and ghtot_ca were made based on V^2/R instead of V*I.
But the main GUI still shows ghptrue
and ghtot based on V*I for now, because it is hoped
the curr3ent readback problem can be fixed in May.
Terminal Bell Volume: From an xterm on the target computer, use the xset
command:
To set bell volume, pitch and duration:
xset
b [vol [pitch [dur]]]
b on
b The b option controls
bell volume, pitch and duration. This
option accepts up to three numerical parameters,
a preceding
dash(-), or a 'on/off' flag. If no parameters are
given, or
the 'on' flag is used, the system defaults will be
used. If
the dash or 'off' are given, the bell will
be turned off. If
only one numerical parameter is given, the bell volume will
be
set to that value, as a percentage of its maximum. Likewise,
the second numerical parameter specifies the
bell pitch, in
hertz, and the third numerical parameter specifies the duration
in milliseconds. Note that not all hardware can vary the bell
characteristics. The X server will set the characteristics of
the bell as closely as it can to the user's specifications.
Transfer Line IV: Look at this link.
It describes usual vacuum (a few mTorr) for the
15/20K coolant lines from the C-can to the target.
GUI_Cleanup: We
had a weird error.
Stipcharts would not come up after a reboot. There
were white alarms on everything that were latched on
but which were neither active nor clearable. But everthing was being read out just fine. Solution was to
kill -9 all processes listed by the "ps -a" command.
Saw made a script called "gui_cleanup" that
should kill all medm, StripTool
and alh processes running under the g0targ
account. It should be enough to type "gui_cleanup".
The script is in /group/gzero/g0targ/bin which is in the path.
Automatic Paging :
April 25, 2003
Steve has successfully set up the alarm paging system.
To turn it on, as cvxwrks@jeffylab goto /home/cvxwrks/PAGER and type
./enable_target_paging.
Likewise, to disable it type
./disable_target_paging.
These little scripts are:
more ./enable_target_paging
#!/bin/sh
cd ~/PAGER/oncall_pager
cp targetdaq.alhConfig
pager.alhConfig
killall alh.new
So the enabler copies the file you edit (targetdaq.alhConfig)
into the
file that is looked at by the alarm handler (pager.alhConfig).
Make
any changes in alarms, etc. to targetdaq.alhConfig
(not to
pager.alhConfig). Or set up new scripts with
different alarm limits
(for parking, for example) and cp them into pager.alhConfig.
This file consists of entries like this:
CHANNEL TARGET hcITC502_3_Sensor_1_R
$ALARMCOUNTFILTER 999999999 10
$STATCOMMAND HIHI pageOnCall.pl -target -grp "TARGET" -sig
"hcITC502_3_Sensor_1_R" -m "hcITC502_
3_Sensor_1_R HIHI"
$STATCOMMAND LOLO pageOnCall.pl -target -grp "TARGET" -sig
"hcITC502_3_Sensor_1_R" -m "hcITC502_
3_Sensor_1_R LOLO"
The 10 on the second line says the alarm condition has to have been on
for 10 seconds in order to generate a page. The actual alarm limit
value is that already in the IOC, defined by the loaded
alarmlimits.bash file in
/u/group/poltar/ctarg/Screens/ALH/alarmlimits.bash.
The kill command kills the pager alarm handler. It is
restarted
automatically by cron. To see that it's in the cron table do a
crontab -l.
Phone numbers for paging are defined in
/home/cvxwrks/PAGER/oncall_pager/pageOnCall.pl
Flatline
Alarm:
Steve W. set up a nice alarm that goes off if the temps have flatlined for 10 successive reads. It runs on cdaql5 under cvxwrks in the file checkbeamAndHVAndTarg.bash in ~/users/willy. See hclog 126947.
In spring 2008 we modified this and put it in a cron job. See hclog 155145.
Beamcurrent
and Raster Alarms:
There is now a GUI which assigns the maximum beam current for each
target we have. The same GUI also allows you to specify whether the raster has
to be on or not for each target. This GUI can be found
at /group/poltar/ctarg/Screens/BDS_maxcurr.adl and you open it from medm.
You can type in the desired beam current maximum for each target. Enter a
raster value of 1000 for those targets you want the raster to be on for. And a zero for those targets
you don't require a raster for. This GUI is only used
to generate alarms through our EPICS alarm handler in the counting house. It doesn't make an FSD or anything like that, just a beep in
the counting house. The alarms themselves in the alarm handler are hcMaxBcmCurrentAlarm and hcRasterMaskAlarm.
These are logical values, they are 0 when the
respective test is OK, and 1 when the test has failed (ie
when the beam current is higher than specified in the BDS_maxcurr.adl
table for the target currently in the beam).
The values used in BDS_maxcurr.adl during spring 2008
are shown here.
To reload all these limits along with all the other alarm limits, click on the
"alarm handler" button in the main GUI, then choose the option
"Reload Alarm Limits". This will load all the values in the file
Gep07.alarms in the /Screens directory.
Turbopump:
The normal vacuum in the scattering chamber is in the 10^-7 Torr
range when the target is cold and cryopumping. The
gate valve interlock which closes the gate valves on both sides of the SC is
set to 5 x 10^-5 Torr- we obviously have to stay well
below this pressure. We don't start a cooldown of the target unless we are well into the 10^-6
range (with the target warm).
If you suspect the scattering chamber turbopump has
tripped off, then you have to go into the hall to check. The turbo controller
for the turbo on the scattering chamber is located in a blue rack, just behind
the target gas panel, behind the right side of the gas panel. You can't miss it, it's about face height, a black box with a
bunch of white stripes running diagonally along the front panel. Normally, the
green power and normal lights are on.There should
only be one or two of the lowest of the diagonal leds
lit, indicating the load is light. If the "failure" light is lit you have to restart it. Press stop, then start. If that
doesn't work, power cycle it and then turn it on and
press start. The load will run up to the top of the diagonal stripes, then back
down over a couple minutes as the turbopump spins up.
The manual is here (second half is in
English).
It can be that the gate valve between the turbo and the scattering chamber has
closed. That valve control is on the 1st floor of the
blue platform (right above your head if you are standing in front of the turbocontroller) about halfway along the long side closest
to the HMS. Just to the left of the 4He panel. The turbo's backing pump is
located right there as well.
There are two cold cathode gauges right next to the turbocontroller.
One reads the vacuum in the girder just upstream of the target. The other reads
the vacuum in the scattering chamber itself.
Power cycling the Fans:
Before you power cycle the fans, you should try to restart the fan from
the GUI. It's finnicky. Try
to clear, reset, off and on. Try it several times. Wait a few seconds between
operations. Try a few permutations on the buttons. Try starting it at 30 Hz
(12%) instead of 60 Hz (25%). Try rebooting the IOC and then retrying to start
from the GUI.
If none of that works, then you will have to power cycle the fans. First, note
the status lights in ALL 3 fan controller GUIs. Having that info in hclog will help us diagnose the proble
later on and should be useful in devising methods to improve reliability of the
fan controllers. Then ask MCC for an ARM, escorted access. Do not wait for a sweep, you need to get the fans back on asap.
Go into Hall C with the ARM. As soon as you're in the
hall, turn right and walk to the row of blue racks just upstream of the green
shielding wall (where the SOS power supplies are). At the nearest end of those
blue racks are some cotton coats that look like they just were
delivered by the dry cleaners. Put one on, this is your PPE for this
activity. There should be safety glasses in the vest pocket. Put those on too.
Then walk to almost the end of the racks. In the blue rack that contains the 3 fan controllers and the three tachs
(waist height) you will find the circuit breaker for the fans. It is mounted on
the side of that rack and it's best to access it from
behind the rack. It's on the right side of the rack,
about chest high, and is the size of a couple of shoe boxes. It's
grey with a big handle on it. Throw the switch on this 480 V circuit breaker
off. Wait 30 seconds, Then turn it back on. Then try
to restart the fan from the GUI.
