RC Meeting and Run Plan, Wednesday, March 4 2020
From HallCWiki
Contents
RC Meeting Information
RC Daily Meetings are M-F at 4:15 PM, 2nd floor Counting House Meeting Room Bluejeans Connection
Wednesday 4 Mar 2020.
General updates
- Target has been running smoothly at around 53%+, but Accelerator has been having problems
- Took ~13 hours of longitudinal data from Tues Swing through Wed Owl
- Accelerator down Wed from 08:30 into swing
- SHMS dipole started having issues during Wed Day shift (see below for details)
- SHMS still offline, as of 5pm experts are in the Hall investigating
- Beam studies took all of Tuesday Day shift and bled into Swing by a couple hours
- Attempt to understand what is going on with the machine (beam loss, activation, etc) [3]
- Identified some significant bunch-length issues on the C beam in particular
- Likely the source of significant beam loss / activation in the machine, and bleedthrough/scraping problems seen in Hall C (and A) during Mollers
- There may be an issue with the Hall C laser (unclear)
- Compressing our bunch with the pre-buncher seems to anti-correlate with getting production quality beam to Hall A
- No easy solution at the moment
- Many thanks to RadCon and OPs for sticking around a little longer while the Target folks worked to wrap up in the Hall
- Target group + RadCon was in the Hall Tuesday Day 10am -- 6pm with work focused on EPR problems
- They had a few issues, but managed to get to a working state by the time they had to get out
- See Hall Access for EPR Debugging and EPR NMR Calibration for more details
- More work during Wed Day shift (during Acc downtime) [4]
- 'Macro Pulse Generator' replacement in injector was attempted Wed Day
- Started 8:30 with a plan to recover by 1:30, but it is running long (still ongoing as of 4pm)
- Attempted installation, but had to back it out later late morning (ran into problems)
- SHMS Dipole has tripped 5 times during Wed day shift [5] [6] [7] [8] [9]
- Steve Lassiter went in the Hall after the 3rd trip (had difficulties resetting remotely) and took a look at the power supply
- Found the 9V power supply that had been replaced earlier had been displaced (mechanical vibration due to dump switches closing?) and corrected that. He felt that was unlikely to be the root cause of the trips however.
- He suspects the V-Loop board may have an issue?
- Jack Segal is investigating whether we have spares and what our near term options are.
- Jack's group is going to take a controlled access and see what they can find.
- Mike Fowler rebooted the Magnet controls IOC in mid-Day shift on Tuesday to correct some issues [10]
- GUI wasn't working quite right, not sure what else may have been affected at that time
- Steve Lassiter went in the Hall after the 3rd trip (had difficulties resetting remotely) and took a look at the power supply
- Joe Did a quick walkthrough training for 2 more people Wed Day
Notes for the Future
- Scheduled Moller measurement for Thursday DAY shift
- 'Filler' runs for next spin up / target-down period (Don't want to forget about this!)
- 1 hour run with target out for measure of the background just from the beryllium windows
Pending Issues
- pNMR issues
- Software/driver issue needs to be resolved
- Checkout during next non-production down time (Moller or SHMS fix)
- EPR problems [11] saw some progress on Tuesday [12], but issues remain
- Tentative plan now is to continue the d2n (and A1n) program by replacing the photo-diode that is placed directly on the pumping chamber (we have many spares)
- It would be good to write up a procedure to streamline this replacement procedure for us and RadCon
- Kepco power supply commissioning/installation (Todd, Bill)
- Bill will update status on Thursday
- Verify that calibration constants and input values associated with NMR and other polarimetry are consistent and will be correct when the software is restarted [13] [14] [15] [16] [17] (Bill, Junhao)
- Persistent BLM & ion chamber trips
- General activation, weird BLM issues (https://logbooks.jlab.org/entry/3792326), Ad-hoc ion chamber threshold adjustments (https://logbooks.jlab.org/entry/3792095)
- NOTE: Shift crew should NOT give authorization to MCC change ion chamber thresholds. If they ask, tell Operator to contact an appropriate expert. Make a log entry if trips persist, or call the RC if it becomes severe (for example, >20 trips/hour is getting intolerable)
- Beam Studies program will investigate some of these issues (most pressing is significant activation + beam loss in the arcs)
- Contact with Target readbacks through AnywhereUSB USB bridge still drop out on occasion
- Initial replacement of the USB bridge failed (did not seem to help underlying issue, then bridge stopped working), replaced with original device
- OPs wished to reboot hung iochc10 [18], noted in December as well [19]
- This runs the Moller cryo controls and should not be rebooted without expert oversight (does not impact operations). D. Gaskell is looking into getting this addressed properly with OPs.
Opportunistic Access Jobs
Target coil rotation planned for March 13--20
- Nominal 1-week downtime in Hall C to rotate target coils, install new cell, and transition to d2n program
- More details are here: PolHe3_Target_Coil_Rotation_Plan,_Mar_2020
Current Run Plan
Wednesday DAY → Thursday OWL
- Beam off around 08:30 for hardware work in Injector -- plan is to have beam restored by 13:30
- When beam goes away around 8:30
- Take longitudinal NMR
- Wait for MCC to return beam (and some guidance on the SHMS dipole problem)...
- Follow the Short beam recovery procedure
- Skip the YAG unless you are concerned about the harp scan data.
- Take longitudinal NMR
- Follow the Short beam recovery procedure
- Continue production data (HMS/SHMS -3.5/-3.4 GeV) with 3He target, longitudinal polarization with NMR every 5 hrs.
Thursday DAY → Friday OWL
NOTE: Plans below may change! Depending on what experts discover about the SHMS dipole problem, the Moller may be deferred in favor of going into Controlled Access for SHMS repairs (or continued investigation).
- Moller measurement planned for Thursday DAY shift
- Just before handing over to Moller team
- Take longitudinal NMR
- Rotate target to 90deg Transverse (Target Expert only)
- Wait for Moller to complete and get production beam re-established (late-DAY, early-SWING)
- Follow the Short beam recovery procedure
- Skip the YAG unless you are concerned about the harp scan data.
- Take transverse NMR
- Take 5 hours of production data (HMS/SHMS -3.5/-3.4 GeV) with 3He target in transverse
- Rotate target to 180deg Longitudinal (Target Expert only)
- Take longitudinal NMR
- Follow the Short beam recovery procedure
- Continue production data (HMS/SHMS -3.5/-3.4 GeV) with 3He target, longitudinal polarization with NMR every 5 hrs.
Standing Orders
If beam goes away for more than 2h, or if during the downtime, MCC does operations that may change the beam tuning, then do [https://hallcweb.jlab.or/wiki/index.php/Beam_and_Raster_Checkout_for_A1n/d2n_(Short_Procedure) beam recovery procedure] (do YAG viewer part only if you are suspicious of the harp scan results)
- NOTE: Shift crew should NOT give authorization to MCC change ion chamber thresholds. If they ask, tell the Operator to contact an appropriate expert. Make a log entry if trips persist, or call the RC if it becomes severe (for example, >20 trips/hour is getting intolerable)
- Check the position locks are on.
- Do your best to accommodate Hall A's request for a PITA scan promptly. It is OK to stop the present runs and start new ones in 5--10 minutes after they are done.
- Check that CODA is running normally (had one strange hangup lately)
- Check online replays shortly after starting runs, & full replay results too.
- Make sure all the beam specs are followed (e.g. no raster size below 4mm on 3He cell).
- Always take 120Hz of EDTM
- Fill BTA with realistic numbers. You may follow the guidance coming from "load from EPICS", if it does not overestimate the ABU (as it generally does). https://logbooks.jlab.org/entry/3783171. If we have to lower the beam current we actually want because the beam is too unstable, prorate the ABUs.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !! !! !! ATTN: SHIFT LEADERS !! !! Please follow the shift summary template on !! !! the laminated SL instructions. !! !! !! !! ATTN: TARGET OPERATORS !! !! All NMR measurement HCLOG entries should include: !! !! 1) A title that clearly lists polarization !! !! 2) Screenshots of both NMR_FieldSweep and !! !! the polarization fit !! !! !! !! For an example see the following entry: !! !! https://logbooks.jlab.org/entry/3768696 !! !! !! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Notices in Effect for Shift Crew
- Policy for CANS-related hall status change: Make a logbook entry when the hall goes in, or comes out of laser-controlled access.
- Magnet policies
- Spectrometer GUI (
go_magnets
) should always be visible on Shift Leader computer, hcdesk3 - HMS Q1 must be ramped down in 50A steps in all cases
- Do not ramp HMS Q2 to zero, only go to 10A and then switch off power.
- If any magnet trips, snapshot the associated power supply window and post it to hclog before resetting the interlock.
- Spectrometer GUI (
- Monitor the beam position on target and call MCC immediately if you see large excursions (1 mm level).
- Ramp rate lock instructions are here and added to checklist[20]
Bluejeans remote connection information
To join the Meeting: https://bluejeans.com/861439920 To join via Room System: Video Conferencing System: bjn.vc -or-199.48.152.152 Meeting ID : 861439920 To join via phone : 1) Dial: +1.408.740.7256 (US (San Jose)) +1.888.240.2560 (US Toll Free)) +1.408.317.9253 (US (Primary, San Jose)) (see all numbers - http://bluejeans.com/numbers) 2) Enter Conference ID : 861439920
Useful Links
Target Polarization Angle Conventions
Be VERY clear here about the convention (beam-right/left/up/downstream) vs 0, 90, 180, 270) for each run period. More notes and references to log entries are better.
Polarization Calibration Data
ADD LIST OF CALIBRATION RUNS HERE AND A TABLE OF RELEVANT CALIBRATION CONSTANTS TO USE FOR WHAT RUNS
Cell | Calibration Type |
SHMS Angle /degree |
SHMS Momentum /GeV |
Field Direction /degree |
Calibration Results | Status | Oven Temperature /°C |
Log# |
---|---|---|---|---|---|---|---|---|
Tommy | EPR-NMR | 11 | 7.5 | 90 | 5.24 %/mV | Done | 200 | 3839939 |
Tommy | EPR-NMR | 18 | 5.6 | 270 | 5.04(1) %/mV | Done | 205 | 3843301 |
Tommy | EPR-NMR | 18 | 5.6 | 90 | undone | 205 | ||
Tommy | EPR-NMR | 11 | 7.5 | 270 | undone | 208 | ||
Tommy | AFP Loss | 14.5 | 6.4 | 90 | 1.25%/Sweep | Done | 200 | 3840618, 3840143, 3840618 |
Tommy | EPR Amplifier on/off NMR Signal |
11 | 7.5 | 90 | Done | 200 | 3839962 | |
Tommy | Temperature Test | 18 | 5.6 | 90 | Done | 205 | 3844089 | |
Tommy | AFP Loss | 18 | 5.6 | 270 | Done | 205 | 3845255 | |
Tommy | Temperature Test | 18 | 5.6 | 270 | Done | 205 | 3845336 | |
Briana | EPR-NMR | 18 | 5.6 | 180 | 8.60 | Done | 205 | 3834036, 3834243 |
Briana | EPR-NMR | 18 | 5.6 | 270 | 5.6 | Done | 205 | 3835512, 3835587 |
Austin | EPR-NMR | 18 | 5.6 | 90 | 4.88 | Done | 200 | 3814717 |
Austin | EPR-NMR | 11 | 7.5 | 180 | 7.64 | Done | 210 | 3817200 |
Austin | EPR-NMR | 11 | 7.5 | 0 | 8.13 | Done | 210 | 3818761 |
Austin | EPR-NMR | 11 | 7.5 | 270 | 4.56 | Done | 210 | 3819882 |
Austin | EPR-NMR | 11 | 7.5 | 270 | 4.58 | Done | 210 | 3822296 |
Austin | EPR-NMR | 11 | 7.5 | 90 | 4.91 | Done | 210 | 3824907 |
Austin | EPR-NMR | 18 | 0 | 90 | 4.58 | Done | 210 | 3815895 |
Austin | EPR-NMR | 11 | 0 | 180 | 6.91 | Done | 210 | 3816724 |
Target Polarization Database
d2n Run Period
D2n Target Run List: https://docs.google.com/spreadsheets/d/16TrIUXVA-znVNUyEKy9FHaIlcvq-KXzu97k9nJowuE8/edit#gid=0
- Junhao Chen is maintaining it
A1n Run Period
- Mingyu Chen will updating the runlist while Murchhana Roy will do the cross check.
Recent polarization measurements
PolHe3 Polarization Measurements
- longitudinal polarization measurements after yesterday's target work in the hall and QWP angle optimization
- Tue 2/18 13:15: 47% https://logbooks.jlab.org/entry/3786686
- Tue 2/18 18:42: 52.2% https://logbooks.jlab.org/entry/3786898
- Tue 2/18 23:47: 54.5% https://logbooks.jlab.org/entry/3787175
- Wed 2/19 4:38: 55.0% https://logbooks.jlab.org/entry/3787175
- Wed 2/19 8:40: 54.9% https://logbooks.jlab.org/entry/3787285
- Wed 2/19 13:08 54.8% https://logbooks.jlab.org/entry/3787420
- Wed 2/19 18:15 54.8% https://logbooks.jlab.org/entry/3787605
- Wed 2/19 11 PM 55.3% https://logbooks.jlab.org/entry/3787781
- Wed 2/19 midnight 51.9% https://logbooks.jlab.org/entry/3787926
- Thu 2/20 05:15 53.2% https://logbooks.jlab.org/entry/3787904
- Thu 2/20 10:34 53.6% https://logbooks.jlab.org/entry/3788085
- Thu 2/20 13:47 53.0% https://logbooks.jlab.org/entry/3788210
- Thu 2/20 21:00 56.0% https://logbooks.jlab.org/entry/3788428
- Fri 2/21 04:07 55.2% L https://logbooks.jlab.org/entry/3788613
- Fri 2/21 09:03 53.8% L https://logbooks.jlab.org/entry/3788795
- Fri 2/21 14:22 52.5% L https://logbooks.jlab.org/entry/3788938
- Fri 2/21 19:50 51.9% L https://logbooks.jlab.org/entry/3789160
- Sat 2/22 01:15 51.6% L https://logbooks.jlab.org/entry/3789331 -> QWP optimization done
- Sat 2/22 06:32 53.0% L https://logbooks.jlab.org/entry/3789442
- Sat 2/22 11:50 54.9% L https://logbooks.jlab.org/entry/3789583 -> rotated to T (5hr) then back to L (2-3hrs)
- Sat 2/22 19:31 43.2% L https://logbooks.jlab.org/entry/3789857
- Sat 2/22 23:08 50.2% L https://logbooks.jlab.org/entry/3789959
- Sun 2/23 04:06 55.3% L https://logbooks.jlab.org/entry/3790104
- Sun 2/23 09:21 55.5% L https://logbooks.jlab.org/entry/3790240
- Sun 2/23 14:17 55.7% L https://logbooks.jlab.org/entry/3790380
- Sun 2/23 19:03 55.7% L https://logbooks.jlab.org/entry/3790513
- Mon 2/24 00:25 55.2% L https://logbooks.jlab.org/entry/3790672
- Mon 2/24 05:47 49.6% L https://logbooks.jlab.org/entry/3790771 -> unexplained drop
- Mon 2/24 10:59 50.8% L https://logbooks.jlab.org/entry/3790965
- Mon 2/24 15:51 50.4% L https://logbooks.jlab.org/entry/3791100
- Mon 2/24 21:22 50.9% L https://logbooks.jlab.org/entry/3791318
- Tue 2/25 02:59 50.7% L https://logbooks.jlab.org/entry/3791481 -> QWP optimization done
- Tue 2/25 08:30 51.4% L https://logbooks.jlab.org/entry/3791660
- Tue 2/25 18:13 ~50% L https://logbooks.jlab.org/entry/3791950
- Tue 2/25 23:13 53.3% L https://logbooks.jlab.org/entry/3792124
- Wed 2/26 04:34 53.6% L https://logbooks.jlab.org/entry/3792291
- Wed 2/26 09:04 54.3% L https://logbooks.jlab.org/entry/3792418 -- NMR just after beam stops
- Wed 2/26 12:46 50.1% L https://logbooks.jlab.org/entry/3792504 -- NMR before beam returns
- Wed 2/26 18:44 52.8% L https://logbooks.jlab.org/entry/3792704
- Wed 2/26 23:42 53.8% L https://logbooks.jlab.org/entry/3792849
- Thu 2/27 04:43 56.4% L https://logbooks.jlab.org/entry/3792931 -- not much beam during owl
- Thu 2/27 08:38 56.3% L https://logbooks.jlab.org/entry/3793085
- Thu 2/27 08:38 56.3% L https://logbooks.jlab.org/entry/3793085
- Thu 2/27 16:50 58.4% L https://logbooks.jlab.org/entry/3793359 -- not much beam during day shift (Moller + retune)
- Thu 2/27 21:49 58.0% L https://logbooks.jlab.org/entry/3793493
- Fri 2/28 02:53 57.8% L https://logbooks.jlab.org/entry/3793610
- Fri 2/28 07:41 57.9% L https://logbooks.jlab.org/entry/3793727
- Fri 2/28 12:54 58.5% L https://logbooks.jlab.org/entry/3793881 -- just before beam restart; after EPR hardware misconfiguration was corrected
- Fri 2/28 18:04 57.2% L https://logbooks.jlab.org/entry/3794035
- Fri 2/28 18:19 49.8% (55.7%) T https://logbooks.jlab.org/entry/3794045 -- value in () is with updated calib. coefficient
- Fri 2/28 20:21 49.2% (55.1%) T https://logbooks.jlab.org/entry/3794079 -- value in () is with updated calib. coefficient
- Sat 2/29 00:37 --- % T https://logbooks.jlab.org/entry/3794242 -- Transverse NMR did not work
- Sat 2/29 01:03 52.5% L https://logbooks.jlab.org/entry/3794244
- Sat 2/29 06:10 54.3% L https://logbooks.jlab.org/entry/3794356
- Some problematic NMR measurements around 12:00--12:14? [21]
- Sat 2/29 12:25 52.8% L https://logbooks.jlab.org/entry/3794512
- Sat 2/29 17:49 54.8% L https://logbooks.jlab.org/entry/3794693
- Sat 2/29 22:51 54.4% L https://logbooks.jlab.org/entry/3794802
- Sun 3/01 03:58 53.9% L https://logbooks.jlab.org/entry/3794944
- Sun 3/01 09:42 54 % L https://logbooks.jlab.org/entry/3795082
- Sun 3/01 11:00 ----- https://logbooks.jlab.org/entry/3795123 -- Re-analyzed some odd sweeps from Sat DAY shift
- Sun 3/01 14:33 54.5% L https://logbooks.jlab.org/entry/3795265
- Sun 3/01 19:58 54.1% L https://logbooks.jlab.org/entry/3795414
- Mon 3/02 01:06 54.1% L https://logbooks.jlab.org/entry/3795560
- Mon 3/02 06:08 53.9% L https://logbooks.jlab.org/entry/3795662
- Mon 3/02 11:08 53.9% L https://logbooks.jlab.org/entry/3795836 -- before switching to Transverse
- Mon 3/02 11:32 52.5% T https://logbooks.jlab.org/entry/3795851
- Mon 3/02 15:04 54.4% T https://logbooks.jlab.org/entry/3795982 -- just before production beam
- Mon 3/02 17:41 https://logbooks.jlab.org/entry/3796117 -- NMR PC Transverse Calibration constant updated
- Mon 3/02 20:37 53.9% T https://logbooks.jlab.org/entry/3796180
- Mon 3/02 20:46 52.9% L https://logbooks.jlab.org/entry/3796182
- Tue 3/03 00:47 55.6% L https://logbooks.jlab.org/entry/3796277
- Tue 3/03 05:36 54.7% L https://logbooks.jlab.org/entry/3796415
- Tue 3/03 08:19 54.7% L https://logbooks.jlab.org/entry/3796485