./monitorDP.sh noticed ANT9_RMS is high and SAT count of 2 BOARD20_STAT of 1024 UTC: 22:07:30 DP INI UTC: 22:13 TBN started at 74.03 added startTBN.sh to MCS/exec Chosen to schedule: LH002, LP001 LH002 Shifting observations to 2012/03/28 07:07:04.091000 UTC LP001 doesn't shift. apparently the RA is greater than 24. abort obs. LH002 shift and moved to MCS/tp/120328/ UTC: 02:00 changed ARX to fullBW TBN changed to 25 MHz Found pointing correction error, deleting corrupt files... Last Recordings on DRs DR1: 056013_000031272 DR2: 056013_000031273 DR3: 056013_000031274 DR4: 056013_000031275 these are... UT 120322 16:30 Hot Jupiter (LH002_90..93) 3 hours LH002_120327_0711_0090_B_HJ.sdf -- Beam 1 -- Recording LH002_120327_0711_0091_B_NS.sdf -- Beam 2 -- Recording LH002_120327_0711_0092_B_HJ.sdf -- Beam 3 -- Recording LH002_120327_0711_0093_B_NS.sdf -- Beam 4 -- Recording DELETED ABOVE Last recordings on DRs DR1: 056010_000015756 DR2: 056010_000015757 DR3: 056008_000012719 DR4: 056010_000015758 these are... LC001 Saturday afternoon - split, DR set 1, LC001_120324_2316_0033_B_JP.sdf -- Beam 1 -- Recording 56010_000015756 LC001_120324_2316_0034_B_T1.sdf -- Beam 2 -- Recording 56010_000015757 LC001_120324_2316_0035_B_SU.sdf -- Beam 4 -- Recording 56010_000015758 and LH002_120322_0731_0087_B_HJ.sdf -- Beam 3 -- Recording 056008_000012718 DELETED ABOVE Last recordings on DRs DR1: 056008_000012717 DR2: 056010_000013171 DR4: 056010_000013172 this is... LH002_120322_0731_0086_B_HJ.sdf -- Beam 1 -- Recording 056008_000012717 Only. DELETE DR1's file, leave the others. UT: 03:50 submitted 4 beams for LP001 (Greg emailed corrected sdf) LH not being run due to pointing correction error -rw-r--r-- 1 op1 op1 1048 2012-03-27 21:44 LP001_120328_1303_0005_B_CA.sdf -rw-r--r-- 1 op1 op1 1052 2012-03-27 21:45 LP001_120328_1303_0006_B_CA.sdf -rw-r--r-- 1 op1 op1 1059 2012-03-27 21:45 LP001_120328_1303_0007_B_CA.sdf -rw-r--r-- 1 op1 op1 1115 2012-03-27 21:45 LP001_120328_1303_0008_B_CY.sdf output: DR1: 056014_000041577_spectrometer.dat DR2: 056014_000041578_spectrometer.dat DR3: 056014_000041579_spectrometer.dat DR4: 056014_000041580_spectrometer.dat 3.00 UT: 03:30 PASI looks like TBN out of sync DP_ INI UT (from PASI): 13:36 VLA scheduled power shutdown, 8 hrs. Above spectrometer files crashed. no MCS tarball. UT: 22:20 power restored. station starting... DR1 SEL 1, last: 056003_000015470 (LC obs, UNM) DR1 SEL 2, last: 056011_000017022 (LK obs, Long Island) DR2 SEL 1, last: 056010_000013171 (DDGBT obs, UNM) DR2 SEL 2, last: 056011_000017023 (LK obs, Long Island) DR3, SEL 1, last: 056003_000015472 (LC obs, UNM) DR3, SEL 2, last: 056011_000017024 (LK obs, Long Island) DR4, SEL 1, last: 056010_000013172 (DDGBT obs, UNM) DR4, SEL 2, last: 056011_000017025 (LK obs, Long Island) selecting UNM drives for all DRs Scheduled SE's pointing runs... at 02:00 03/29/12 -f 120329_CygA_0802.sh at 07:39 03/29/12 -f 120329_CygA_1341.sh at 09:40 03/29/12 -f 120329_CygA_1542.sh at 11:26 03/29/12 -f 120329_CygA_1728.sh at 17:13 03/29/12 -f 120329_TauA_2315.sh DR remaining storage... DR1: 7181.22 GB DR2: 4771.65 GB DR3: 7181.22 GB DR4: 4137.75 GB LR001 session 2 will be 1 TB (one beam) LH001 are 800 GB per beam (all 4 beams) Running SE pointing drift scripts (spectrometer) until 03/30/12, 21:11 (MDT) UTC: 3/30/12, 00:00 at 21:15 03/30/12 -f setARX_full.sh at 21:16 03/30/12 -f INIdp.sh LK001 (3/30/12 - 21:40) OBS_START UTC 2012 03 31 03:40:28.637 OBS_DUR+ 06:00:00.000 only 3 beams running, shift bug and SESSION_DRX_BEAM was not specfied in SDF. Looks like MCS sorted it out and is recording and BAMing on 3 beams. Last Recordings... DR1: 056017_000014932 DR2: 056017_000014930 DR3: 056017_000014931 DR4: 056011_000017025 at 4:30 03/31/12 -f setARX_split.sh at 4:35 03/31/12 -f startTBN_split74.sh PASI at 21:00 03/31/12 -f setARX_full.sh at 21:05 03/31/12 -f INIdp.sh LK001 (3/31/12 - 21:40) OBS_START UTC 2012 04 01 03:36:28.637 OBS_DUR+ 06:00:00.000 Last Recordings... DR1: 056018_000002659 DR2: 056018_000002660 DR3: 056018_000002661 DR4: 056018_000002662 remaining storage on Long Island DRSUs... DR1: 1284.40 GB DR2: 1284.40 GB DR3: 1284.40 GB DR4: 2890.53 GB DRSU SEL 1 (UNM) remaining storage on UNM DRSUs... DR1: 7181.22 GB DR2: 4771.65 GB DR3: 7181.22 GB DR4: 4137.75 GB UT: 12/04/01 13:00, split BW DDGBT (4/1/12 - 9:00), run beams 1 & 3 OBS_START UTC 2012 04 01 15:00:00.000 OBS_DUR+ 06:00:00.000 Last Recording... DR1: 056018_000007541 Recorded: 1590.19 GB reboot DR3 DR3: 056018_000002661(SEL 1, but this is Long Island file) unable to access UNMs DRSU (see "DR3 timeout / Asynchronous write" exploder thread) at 15:15 04/01/12 -f startTBN_split74.sh at 20:30 04/01/12 -f INIdp.sh NOTE: strong RFI seen in PASI, multiple freqs. LR001 (beam 4) 20:56 MT OBS_START UTC 2012 04 02 02:56:04.091 OBS_DUR+ 04:00:00.000 NOTE: around 4:00 UT, beam 3 underflowed DP INI at 04:11 UT DRX's issues around 04:37 UT restarted DRs 1, 2, 3 around 14:00 UT (timeout responses, no spectrometer files created) COMM runs OBS_DUR+ 04:00:00.000 S PROJ SESS B MJD MPM DUR NOBS C 0 4 DDGBT 3 1 56018 53995000 21610000 1 0 DDGBT_0003.ses 1 4 DDGBT 4 3 56018 53995000 21610000 1 0 DDGBT_0004.ses 2 3 COMM 40 1 56019 14275000 14050000 1 0 COMM_0040.ses 3 3 COMM 41 2 56019 14275000 14050000 1 0 COMM_0041.ses 4 3 COMM 51 1 56019 75535000 21610000 1 0 COMM_0051.ses 5 3 COMM 52 2 56019 75535000 21610000 1 0 COMM_0052.ses 6 3 COMM 53 3 56019 75535000 21610000 1 0 COMM_0053.ses 7 3 COMM 54 4 56019 75535000 21610000 1 0 COMM_0054.ses 8 3 COMM 42 3 56019 14275000 14050000 1 0 COMM_0042.ses 9 3 LR001 2 4 56019 10559091 14410000 1 0 LR001_0002.ses op1@mcs-tp:~/MCS/tp/mbox$ tail -10000 mesq.dat S PROJ SESS B MJD MPM DUR NOBS C 0 3 COMM 62 2 56020 14035000 14050000 1 0 COMM_0062.ses 1 3 COMM 63 3 56020 14035000 14050000 1 0 COMM_0063.ses 2 3 COMM 64 4 56020 14035000 14050000 1 0 COMM_0064.ses 3 3 COMM 61 1 56020 14035000 14050000 1 0 COMM_0061.ses S PROJ SESS B MJD MPM DUR NOBS C 0 3 COMM 71 1 56020 75235000 21610000 1 0 COMM_0071.ses 1 3 COMM 72 2 56020 75235000 21610000 1 0 COMM_0072.ses 2 3 COMM 73 3 56020 75235000 21610000 1 0 COMM_0073.ses 3 3 COMM 74 4 56020 75235000 21610000 1 0 COMM_0074.ses 4 3 COMM 81 1 56021 13795000 14050000 1 0 COMM_0081.ses 5 3 COMM 82 2 56021 13795000 14050000 1 0 COMM_0082.ses 6 3 COMM 83 3 56021 13795000 14050000 1 0 COMM_0083.ses 7 3 COMM 84 4 56021 13795000 14050000 1 0 COMM_0084.ses * don't record raw to beam 3, only Long Island DRSU showing up * UT date UT time Project Duration (Hrs) N_beams mode stat 120403 2055 COMM 6 4 spec done 120404 0255 free 1 120404 0350 COMM 4 4 spec done 120404 0753 COMM 2 4 spec done 120404 1000 free 5 120404 1522 LH001 4 1 (2) raw done 120404 1930 free (TEST) 6 INI at 20:00 04/04/12 -f INIdp.sh 120405 0244 LR001 4 1 (4) raw done full at 00:44 04/05/12 -f setARX_full.sh 120405 0645 LH002 3 3 (1,2,4) raw done 120405 0945 free 2 120405 1231 LP001 3 3 (1,2,4) raw done split at 10:00 04/05/12 -f setARX_split.sh TBN at 10:01 04/05/12 -f startTBN_split74.sh 120405 1600 free (LH1 tst) 6 120405 2245 COMM 2 4 spec running 120406 0100 free 2 120406 0230 LR001 4 1 (1) raw sch full at 00:38 04/06/12 -f setARX_full.sh 120406 0640 LH002 3 3 (1,2,4) raw sch 120406 1000 free 3 120406 1311 COMM 2 4 spec sch 120406 1600 collection 4 (UT 04/05/12 22:45, 2 hrs) at 16:45 04/05/12 -f 120405_TauA_2247.sh (UT 04/06/12 13:11 at 07:11 04/06/12 -f 120406_CygA_1313.sh 120404 17:00 UT, remaining storage (bytes) DR1: 5986211332096 DR2: 3973229051904 DR4: 3292897738752 CasA UC: 17:40 UT CygA transit: 14:13 UT 1 3 3 3 3 3 56022 24295000 1 10810000 0 LH002 94 56022 24295000 2 10810000 0 LH002 95 56022 24295000 4 10810000 0 LH002 97 56022 45087728 1 10810000 0 LP001 9 56022 45087728 2 10810000 0 LP001 10 56022 45087728 4 10810000 0 LP001 11 56022 9851364 4 14410000 0 LR001 3 120405 02:44:10 56022 9850365 T 41584 1 DR4 REC 56022 9856364 14410000 DRX_FILT_7| 120405 02:44:10 56022 9850766 T 41598 1 DR4 RPT SUMMARY| 120405 02:44:10 56022 9850766 T 41598 2 DR4 RPT SUMMARY| 120405 02:44:10 56022 9850766 T 41599 1 DR4 RPT OP-TYPE| 120405 02:44:10 56022 9850767 T 41599 2 DR4 RPT OP-TYPE| 120405 02:44:11 56022 9851363 T 41584 2 DR4 REC 56022 9856364 14410000 DRX_FILT_7| 120405 02:44:14 56022 9854913 T 41598 3 DR4 RPT NORMAL | 120405 02:44:19 56022 9859920 T 41599 3 DR4 RPT Idle | 120405 02:44:20 56022 9860076 T 41584 6 DR4 REC Invalid time| try DR4 REC "56022 9900000 12600000 DRX_FILT_7" 120405 03:04:56 56022 11096172 T 41986 1 DR4 REC 56022 11100000 12600000 DRX_FILT_7| 120405 03:04:56 56022 11096172 T 41986 2 DR4 REC 56022 11100000 12600000 DRX_FILT_7| 120405 03:04:56 56022 11096644 T 41986 3 DR4 REC 056022_000041986| started at 03:05 UT, for 3.5 hours, done at 06:35 UT now 56022 9853365 SC LR001 3 4 0 me_action(): cmd: 56022 9854364 0 DP_ DRX 20 '4 1 38000000 7 4 36' 56022 9853367 SC LR001 3 4 0 me_action(): cmd: 56022 9854364 0 DP_ DRX 20 '4 2 57600000 7 4 36' Reference #41603: Final State: 6 Date: 120405 02:44:13 Beam: 4 Tuning: 1 Frequency: 38000000.000 Bandwidth: 7 Gain: 4 Subslot: 36 Reference #41604: Final State: 6 Date: 120405 02:44:13 Beam: 4 Tuning: 2 Frequency: 57600000.000 Bandwidth: 7 Gain: 4 Subslot: 36 Remaining Storage: DR1: 5035460132864 (UNM) DR2: 3022477852672 (UNM) DR3: 1379115204608 (Long Island) DR4: 1336613011456 (UNM) outrigger dipole on DR1 beam on DR2 Remaining Storage DR1: 4460869320704 DR2: 1872179494912 DR4: 1336613011456 UT: 120406, 13:25 Remaining Storage DR1: 2448206331904 DR2: 1009495572480 DR4: 473929089024 Data Collected, all DRSUs. New DRSUs... DR1: S10TCC13S0013 (md1) - Virginia Tech DRSU-3: sdh sdi sdj sdk sdl S10TCC13S0023 (md0) - LWA: sda sdb sdc sdd sde DR2: S10TCC13S0010 (md0) - Virginia Tech DRSU-5: sdg sdh sdi sdj sdk S10TCC13S0014 (md1) - LWA: sdb sdc sdd sde sdf DR3: S10TCC13S0008 (md1) - LWA: sdb sdc sdd sde sdf S10TCC13S0012 (md0) - Virginia Tech DRSU-2: sdg sdh sdi sdj sdk DR4: S10TCC13S0011 (md1) - Virginia Tech DRSU-4: sda sdb sdc sdd sde S10TCC13S0021 (md0) - LWA: sdh sdi sdj sdk sdl DR5: S10TCC13S0022 (md0) - LWA: sdb sdc sdd sde sdf DR1 SEL 1 = md0 (LWA) SEL 2 - md1 (VT) DR2 SEL 1 = md0 (VT) SEL 2 - md1 (LWA) DR3 SEL 1 = md0 (VT) SEL 2 - md1 (LWA) DR4 SEL 1 = md0 (LWA) SEL 2 - md1 (VT) DRX commands timeout for LE002 sessions. Large loss of data. Deleted files on DR3 & DR4, but didn't delete DR1/2 because the next session started before I could do it. We will shift a rerun the LE002 session on 120409. DRX 48386 48387 Reference #48386: Final State: 6 Date: 120405 16:59:57 Beam: 2 Tuning: 1 Frequency: 38000000.000 Bandwidth: 7 Gain: 4 Subslot: 0 Reference #48387: Final State: 6 Date: 120405 16:59:57 Beam: 2 Tuning: 2 Frequency: 68000000.000 Bandwidth: 7 Gain: 4 Subslot: 0 ./mesix DP_ DRX "2 1 38000000 7 4 0" INI at UT 20:00 at UT 120409 03:13, exec crashed with "Insufficient advance notice" restart exec at 04:00 BAMs not happening. noted "beam not repointing" in the op-log spreadsheet. Hand-off to F. Schinzel, 16:00 UT.