1500 Y Pihlstrom taking over for G Taylor DR1 DRSUs Attached: 2 DRSU1: /dev/md0 DRSU2: /dev/md1 Selected DRSU: 1 Total Storage: 9315.08 GB Remianing Storange: 6227.08 GB Usage: 33.00% DR2 DRSUs Attached: 2 DRSU1: /dev/md0 DRSU2: /dev/md1 Selected DRSU: 1 Total Storage: 9315.08 GB Remianing Storange: 4046.27 GB Usage: 56.00% DR3 DRSUs Attached: 2 DRSU1: /dev/md0 DRSU2: /dev/md1 Selected DRSU: 1 Total Storage: 9315.08 GB Remianing Storange: 3986.55 GB Usage: 57.00% DR4 DRSUs Attached: 2 DRSU1: /dev/md0 DRSU2: /dev/md1 Selected DRSU: 2 Total Storage: 9315.08 GB Remianing Storange: 3920.29 GB Usage: 57.00% 2000 Restarted TBN 0030 LP001_120508_0400_0034_B2_TS.sdf submitted DRX_FILT_4, GAIN 5 0030 LP001_120508_0400_0035_B3_TS.sdf submitted DRX_FILT_4, GAIN 5 0030 LP001_120508_0400_0036_B4_TS.sdf submitted DRX_FILT_4, GAIN 5 5 min test observations to check gain setting 1400 Beam 1 underflow 1400 Restarted DP_ INI 1400 Restarted TBN 1445 Beam 1 MGT underflow 1446 DP_ INI 1447 Restarted TBN 1800 Beam 1 MGT underflow 1820 DP_ INI 1825 Restarted TBN 2015 LP001_120508_2015_0037_B2_TS.sdf submitted DRX_FILT_4, GAIN 4 2015 LP001_120508_2015_0038_B3_TS.sdf submitted DRX_FILT_4, GAIN 4 2015 LP001_120508_2015_0039_B4_TS.sdf submitted DRX_FILT_4, GAIN 4 5 min test observations to test gain settings Looked at the six files with the different gain settings, gain 4 uses about 6-7 bits versus the 4 bits for gain 5 so will use gain 4 in the next set of observations. 2330 Estimate Power failure at site, all system down 0130 Power back on, starting up Not working correctly, there are board errors and antenna rms values all 0 0200 Restarting MCS-DP software due to BOARD errors, DP won't recognize the boards 0230 Jayce helps to restart properly 1130 Restart TBN 1800 Software update by Jayce 2200 DP and ASP shutdown because of HVAC iced up. 2145 Station back after HVAC fixed and Jayce's testing 2153 Start TBN back up 2154 setARX_split.sh (0/15/8/6) 2349 DP_ INI due to all beams beam former calibration failed 0000 setARX filter to split 0030 restarted DP_ INI and TBN several times due to Beam 1 underflow 0100 LR001_120511_0100_0013_B2_PS.sdf, Paul's 09hLST pulsar monitoring, gain 7 0246 Beam 1 underflow, DP_ INI and restarted TBN 0944 Beam 1 underflow, DP_ INI and restarted TBN 1100 LR001_120511_1100_0014_B2_PS.sdf, Paul's 19hLST pulsar monitoring, gain 7 1230 LP001_120511_1230_0040_B2_CA.sdf, gain 4 (bw setting 4, 2MHz) 1230 LP001_120511_1230_0041_B3_CA.sdf, gain 4 (bw setting 4, 2MHz) 1230 LP001_120511_1230_0042_B4_CA.sdf, gain 4 (bw setting 4, 2MHz) 1545 Hand over station to Jayce for software tests 2300 Startup procedures run, start TBN 1155 Beam 1 underflow, DP_ INI and restart TBN 1215 Beam 1 underflow, DP_ INI and restart TBN 1230 LP001_120512_1230_0043_B2_CA.sdf, gain 4 (bw setting 4, 2MHz) 1230 LP001_120512_1230_0044_B3_CA.sdf, gain 4 (bw setting 4, 2MHz) 1230 LP001_120512_1230_0045_B4_CA.sdf, gain 4 (bw setting 4, 2MHz) 2320 DP_ INI restart TBN 1200 LP001_120513_1200_0046_B2_CA.sdf, gain 4 (bw setting 4, 2MHz) 1200 LP001_120513_1200_0047_B3_CA.sdf, gain 4 (bw setting 4, 2MHz) 1200 LP001_120513_1200_0048_B4_CA.sdf, gain 4 (bw setting 4, 2MHz) 2141 Beam 1 underflow, DP_ INI and restart TBN 2316 Beam 1 underflow, DP_ INI and restart TBN 0100 LR001_120514_0100_0015_B2_PS.sdf, gain 7 (bw setting 7) pulsar 9h LST monitoring 1100 LR001_120514_1100_0016_B2_PS.sdf, gain 7 (bw setting 7) Pulsar 19h LST monitoring 1230 LP001_120514_1230_0049_B2_CA.sdf, gain 4 (bw setting 4, 2MHz) 1230 LP001_120514_1230_0050_B3_CA.sdf, gain 4 (bw setting 4, 2MHz) 1230 LP001_120514_1230_0051_B4_CA.sdf, gain 4 (bw setting 4, 2MHz) 1705 Handing over to Chris ---- C Wolfe taking over from Y Pihlstrom 1730 logged into mcstp, while reviewing startup procedure, noticed station temps climbing as if AC is off. 1800 station start script run: DP shows MGT underflow, PASI/TBN still running 1929 Joe killed ASP/DP, restarted MCS. Station Temps returned to 67 from 80. HVAC shut off, temps started climbing again. 2200 ASP/DP powered up, HVAC seems to be holding. 2350 LK001_120515_0000_11_B1_NS.sdf bw. 7, gain default 2350 LK001_120515_0000_11_B2_NS.sdf bw. 7, gain default 2350 LK001_120515_0000_11_B3_SP.sdf bw. 7, gain default 2350 LK001_120515_0000_11_B4_SP.sdf bw. 7, gain default 0010 LK001 observations failed : dropped messages + DR1 out of sync (ntp was stopped). rescheduling for 0200 UTC 0129 submitted LK001_120515_0200_11_B1_NS.sdf bw. 7, gain default 0129 submitted LK001_120515_0200_11_B2_NS.sdf bw. 7, gain default 0129 submitted LK001_120515_0200_11_B3_SP.sdf bw. 7, gain default 0129 submitted LK001_120515_0200_11_B4_SP.sdf bw. 7, gain default 0321 noticed current R3 slightly high (17), but all else seems to be in order. continuing on. 0327-0430 MGT underflow / beamformer calibration lost. Terminated recordings, restarted executive+DP 0743 submitted LP001_120515_1130_52_B2_CA.sdf bw. 7, gain default 0743 submitted LP001_120515_1130_53_B3_CA.sdf bw. 7, gain default 0743 submitted LP001_120515_1130_54_B4_CA.sdf bw. 7, gain default 1130 started LP001_120515_1130_52_B2_CA.sdf bw. 7, gain default 1130 started LP001_120515_1130_53_B3_CA.sdf bw. 7, gain default 1130 started LP001_120515_1130_54_B4_CA.sdf bw. 7, gain default 1421 DP_ RPT T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected Only beam 1, continuing with observation 1429 finished LP001_120515_1130_52_B2_CA.sdf bw. 7, gain default 056062_000088362 1429 finished LP001_120515_1130_53_B3_CA.sdf bw. 7, gain default 056062_000088363 1429 finished LP001_120515_1130_54_B4_CA.sdf bw. 7, gain default 056062_000088364 ???? Joe INI'd DP to shut off beams due to thermals. 1939 shut down exec for port-swap 1940 shut down scheduler for port-swap 2000 restart exec/scheduler after port swap. No issues. 2121 shut down DP_ and ASP for a couple hours to allow HVAC to deice 0122 restart ASP/DP , started TBN 0130 PASI garbled, restart ASP/DP reconfig'd ASP for split BW, started TBN 0400-0500 restarted DP several times while debugging script 'restartDP.sh' 0518 debugging complete, restarted TBN 0743 lost cal. (MGT underflow), restarted DP 0747 DP back up, restarted TBN 0820 submitted Dipole-Dipole fringe observation via: "at 05:00 05/16/12 -f 120516_CygA_0502.sh" 1517 confirmed Dipole-Dipole fringe observation: No warnings/errors OP_TAG: 056063_000008630 1600 restarted DP to turn beams off 1605 restarted TBN, split 74 1610 calibration lost, restarting DP/TBN 1707 system down'd for planned maintenance 1855 system up'd, started TBN 74MHz splitBW ASP 2302 submitted LR001_120517_0100_0017_B2_PS.sdf bw 7 0059 calibration lost on beam 1. continuing with scheduled observation (0100) ... 056064_000001823 0200 LR001_120517_0100_0017_B2_PS.sdf bw 7 completed. 0329 restarted DP, TBN, split 74 0344 calibration lost on beam 1. No operations scheduled 0345 restarted DP, TBN, split 74 0405 submitted /LR001_120517_1100_0018_B2_PS.sdf gain7, split BW 0530 calibration lost on beam 1. LR001_120517_1100_0018_B2_PS scheduled 0530 restarted DP, TBN, split 74 ???? calibration lost on beam 1 (MGT underflow) 0849 restarted DP, TBN, split 74 1100 /LR001_120517_1100_0018_B2_PS.sdf gain7, split BW started as scheduled 1342 calibration lost on beam 1 restarted DP, TBN, split 74 1941 calibration lost on beam 1 restarted DP, TBN, split 74 0000 noticed PASI stuck, sent email to staff exploder 0624 submitted LP001_120518_1000_0055_B2_CA.sdf gain7, split BW 0624 submitted LP001_120518_1000_0056_B3_CA.sdf gain7, split BW 0624 submitted LP001_120518_1000_0057_B4_CA.sdf gain7, split BW 1000 LP001_120518_1000_0055_B2_CA.sdf gain7, split BW started on schedule 1000 LP001_120518_1000_0056_B3_CA.sdf gain7, split BW started on schedule 1000 LP001_120518_1000_0057_B4_CA.sdf gain7, split BW started on schedule 1300 LP001_120518_1000_0055_B2_CA.sdf gain7, split BW completed on schedule 1300 LP001_120518_1000_0056_B3_CA.sdf gain7, split BW completed on schedule 1300 LP001_120518_1000_0057_B4_CA.sdf gain7, split BW completed on schedule 0030 PASI restarted 0629 submitted LP001_120518_1000_0058_B2_CA.sdf gain7, split BW 0629 submitted LP001_120518_1000_0059_B3_CA.sdf gain7, split BW 0629 submitted LP001_120518_1000_0060_B4_CA.sdf gain7, split BW 1000 LP001_120518_1000_0058_B2_CA.sdf started as scheduled 1000 LP001_120518_1000_0059_B3_CA.sdf started as scheduled 1000 LP001_120518_1000_0060_B4_CA.sdf started as scheduled 1029 MGT underflow on beam 1, calibration lost 1300 LP001_120518_1000_0058_B2_CA.sdf completed as scheduled 1300 LP001_120518_1000_0059_B3_CA.sdf completed as scheduled 1300 LP001_120518_1000_0060_B4_CA.sdf completed as scheduled 2114 restarted dp,tbn split 74 2118 submitted LR001_120520_0115_0019_B2_PS.sdf gain 7, split BW 2120 FPGA temperatures stopped updating, didn't notice until ~2300 0110 Issued DP INI, DP remained "BOOTING", and would not return to "NORMAL" Meanwhile, DP was still responding to and accepting BAM/DRX commands, so data for LR001_120520_0115_0019_B2_PS.sdf may be ok. 0115 LR001_120520_0115_0019_B2_PS.sdf gain 7, split BW started. 0230 LR001_120520_0115_0019_B2_PS.sdf gain 7, split BW completed. 0304 logged into DP-MCS and restarted DP-MCS as per operator's manual, page 13, item 3 DP SUMMARY now reports NORMAL and FPGA temperatures are also being updated on the operations website. 0755 submitted LP001_120520_0900_0061_B2_CA.sdf, gain 7, split BW 0755 submitted LP001_120520_0900_0062_B3_CA.sdf, gain 7, split BW 0756 submitted LP001_120520_0900_0063_B4_CA.sdf, gain 7, split BW 0759 submitted LR001_120520_1215_0020_B2_PS.sdf, gain 7, split BW 0850 Noticed Beam 1 calibration lost, not enough time to INI DP before next obs. 0900 LP001_120520_0900_0061_B2_CA.sdf, gain 7, split BW started as scheduled 0900 LP001_120520_0900_0062_B3_CA.sdf, gain 7, split BW started as scheduled 0900 LP001_120520_0900_0063_B4_CA.sdf, gain 7, split BW started as scheduled 1200 LP001_120520_0900_0061_B2_CA.sdf completed as scheduled 1200 LP001_120520_0900_0062_B3_CA.sdf completed as scheduled 1200 LP001_120520_0900_0063_B4_CA.sdf completed as scheduled 1530 manually started EPO_120520_1530_0001_TN_XX.sdf 1600 submitted LW001_120520_2330_0005_B1_SU.sdf, gain 7, full BW 1600 submitted LW001_120520_2330_0008_B4_NCP.sdf, gain 7, full BW 1600 submitted LW001_120520_2330_0006_B2_SU.sdf, gain 7, full BW 1600 submitted LW001_120520_2330_0007_B3_SU.sdf, gain 7, full BW 2050 restarted executive, resubmitted corrected jobs 2059 resubmitted LW001_120520_2330_0008_B4_NCP.sdf, gain 7, full BW 2059 resubmitted LW001_120520_2330_0006_B2_SU.sdf, gain 7, full BW 2059 resubmitted LW001_120520_2330_0007_B3_SU.sdf, gain 7, full BW 2059 resubmitted LW001_120520_2330_0005_B1_SU.sdf, gain 7, full BW 2220 EPO_120520_1530_0001_TN_XX.sdf completed, likely corrupted because DR5 stopped responding, runtime log shows several : "[FILE SYSTEM] Error: Asynchronous write reported an error. -1" 2230 LW001_120520_2330_0005_B1_SU.sdf, gain 7, full BW started as scheduled 2230 LW001_120520_2330_0008_B4_NCP.sdf, gain 7, full BW started as scheduled 2230 LW001_120520_2330_0006_B2_SU.sdf, gain 7, full BW started as scheduled 2230 LW001_120520_2330_0007_B3_SU.sdf, gain 7, full BW started as scheduled 0200 LW001_120520_2330_0005_B1_SU.sdf, gain 7, full BW completed as scheduled 0200 LW001_120520_2330_0006_B2_SU.sdf, gain 7, full BW completed as scheduled 0200 LW001_120520_2330_0007_B3_SU.sdf, gain 7, full BW completed as scheduled 0200 LW001_120520_2330_0008_B4_NCP.sdf, gain 7, full BW completed as scheduled 0612 submitted LP001_120521_1000_0064_B2_CA, gain 7, split BW 0613 submitted LP001_120521_1000_0065_B3_CA, gain 7, split BW 0620 submitted LP001_120521_1000_0066_B4_CA, gain 7, split BW 1000 LP001_120521_1000_0064_B2_CA, gain 7, split BW started as scheduled 1000 LP001_120521_1000_0065_B3_CA, gain 7, split BW started as scheduled 1000 LP001_120521_1000_0066_B4_CA, gain 7, split BW started as scheduled 1300 LP001_120521_1000_0064_B2_CA, gain 7, split BW completed as scheduled 1300 LP001_120521_1000_0065_B3_CA, gain 7, split BW completed as scheduled 1300 LP001_120521_1000_0066_B4_CA, gain 7, split BW completed as scheduled 1813 restarted DP, TBN, split BW 74 MHz 2000 Power-sycled DRSUs 4 and 5, rebooted DR4, DR5 2020 DR4, DR% are both up, responsive, and show expected DRSUs and files =========== Storage report for DR1 =========== DR1 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 13972.59 GB Remaining Storage: 13835.87 GB Contiguous Storage: 13835.87 GB Usage: 0% File Count: 1 files DRSU2: /dev/md1 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 9315.05 GB Contiguous Storage: 9315.08 GB Usage: 0% File Count: 0 files =========== Storage report for DR2 =========== DR2 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 13972.59 GB Remaining Storage: 11767.82 GB Contiguous Storage: 11767.82 GB Usage: 15.00% File Count: 9 files DRSU2: /dev/md1 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 9315.05 GB Contiguous Storage: 9315.08 GB Usage: 0% File Count: 0 files =========== Storage report for DR3 =========== DR3 DRSUs Attached: 2 DRSU selected: 2 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 9315.05 GB Contiguous Storage: 9315.08 GB Usage: 0% File Count: 0 files DRSU2: /dev/md1 Barcode: '<Not Implemented Yet>' Total Storage: 13972.59 GB Remaining Storage: 12974.83 GB Contiguous Storage: 12974.83 GB Usage: 7.00% File Count: 5 files =========== Storage report for DR4 =========== DR4 DRSUs Attached: 2 DRSU selected: 2 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 9315.05 GB Contiguous Storage: 9315.08 GB Usage: 0% File Count: 0 files DRSU2: /dev/md1 Barcode: '<Not Implemented Yet>' Total Storage: 13972.59 GB Remaining Storage: 12974.83 GB Contiguous Storage: 12974.83 GB Usage: 7.00% File Count: 5 files =========== Storage report for DR5 =========== DR5 DRSUs Attached: 1 DRSU selected: 1 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 6199.83 GB Contiguous Storage: 6199.83 GB Usage: 33.00% File Count: 1 files 2030 Chris Wolfe signing off, Handing over station control to Jake Hartman 2330 Chris Wolfe retaking the helm until noon EDT 2353 submit LR001_120522_1100_0021_B2_PS.sdf gain 7, split BW 2354 planned LR001_120523_0100_0021_B2_PS.sdf gain 7, split BW Submitting deferred until Jake takes over. SDF is stored in ~/MCS/tp/120523/Planning/LR001_120523_0100_0021_B2_PS.sdf 0010 queued restartDP.sh using "at" to be run at 0430 MDT (1030 UTC) 0030 queued postObservationActions.sh using "at" to be run at 0615 MDT (1215 UTC) 0410 restarted DP/TBN 0430 restart DP, no TBN, by way of "at" command 0500 LR001_120522_1100_0021_B2_PS.sdf gain 7, split BW started as planned 0600 LR001_120522_1100_0021_B2_PS.sdf gain 7, split BW completed as planned, MGT underflow on beam 1 0630 postObservationActions.sh executed, by way of "at" command 0830 Temperatures seem to be making slow climb consistent with HVAC iceing, but too early to tell. 0830 Ran storage report =========== Storage report for DR1 =========== DR1 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 13972.59 GB Remaining Storage: 13835.87 GB Contiguous Storage: 13835.87 GB Usage: 0% File Count: 1 files DRSU2: /dev/md1 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 9315.05 GB Contiguous Storage: 9315.08 GB Usage: 0% File Count: 0 files =========== Storage report for DR2 =========== DR2 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 13972.59 GB Remaining Storage: 11499.51 GB Contiguous Storage: 11499.51 GB Usage: 17.00% File Count: 10 files DRSU2: /dev/md1 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 9315.05 GB Contiguous Storage: 9315.08 GB Usage: 0% File Count: 0 files =========== Storage report for DR3 =========== DR3 DRSUs Attached: 2 DRSU selected: 2 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 9315.05 GB Contiguous Storage: 9315.08 GB Usage: 0% File Count: 0 files DRSU2: /dev/md1 Barcode: '<Not Implemented Yet>' Total Storage: 13972.59 GB Remaining Storage: 12974.83 GB Contiguous Storage: 12974.83 GB Usage: 7.00% File Count: 5 files =========== Storage report for DR4 =========== DR4 DRSUs Attached: 2 DRSU selected: 2 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 9315.05 GB Contiguous Storage: 9315.08 GB Usage: 0% File Count: 0 files DRSU2: /dev/md1 Barcode: '<Not Implemented Yet>' Total Storage: 13972.59 GB Remaining Storage: 12974.83 GB Contiguous Storage: 12974.83 GB Usage: 7.00% File Count: 5 files =========== Storage report for DR5 =========== DR5 DRSUs Attached: 1 DRSU selected: 1 DRSU1: /dev/md0 Barcode: '<Not Implemented Yet>' Total Storage: 9315.08 GB Remaining Storage: 6199.83 GB Contiguous Storage: 6199.83 GB Usage: 33.00% File Count: 1 files 1130 Chri Wolfe, siging out, handing over station control to Jake Hartman