UT 120229 13:01 All DRs are reporting OP-TYPE = Idle UT 120229 13:05 DRSU-COUNT is 1,2,2,2,1 for DRs 1,2,3,4,5 respectively UT 120229 13:06 DRSU-SELECTED is 1,2,1,1,1 for DRs 1,2,3,4,5 respectively UT 120229 13:39 Logged contents of USB ext HDDs attached to DRs 2-5. UT 120229 14:22 Completed inventory of files on DRSUs UT 120229 17:00 FS is at site. Powered down all DRs so he can rearrange things. "$ shutdown -g0" strange for DR2-5. Get shutdown message, get kicked out, can't log back in, but still responding to "$ ping" UT 120229 19:31 DRs 1-4 now all have 2 DRSUs, DR5 has 1. FS has left site. UT 120229 19:31 Suddenly getting "Component not available <DRSU>" error on DR1. Actually, both DRs. Was able to use it moments ago... UT 120229 19:31 Doing DR1 INI in hope that it clears above problem UT 120229 19:36 Looks like WAVE_Y on DR1 crashed -- not showing up in top; not responding via MCS UT 120229 19:38 "$ sudo reboot" on DR1 -- that fixed everything UT 120229 19:40 Copying 2 hours worth to /media/caldbeck: (later deleted from DRSU) $ ./mesix DR1 CPY "055981_000021047 0 568889893192 /dev/sdg1 120224_0500_B1_NS.dat" UT 120229 19:49 Copying 2 hours worth to /media/henderson: (later deleted from DRSU) $ ./mesix DR3 CPY "055981_000021048 0 568889893192 /dev/sdg1 120224_0500_B2_NS.dat" UT 120229 20:04 Copying 2 hours worth to /media/anders: (later deleted from DRSU) $ ./mesix DR4 CPY "055981_000021072 0 568889893192 /dev/sdd1 120224_0500_B3_SP.dat" UT 120301 02:40 Copying 2 hours worth to /media/caldbeck: (later deleted from DRSU) $ ./mesix DR1 CPY "055982_000027272 0 568889893192 /dev/sdg1 120225_0500_B1_NS.dat" UT 120301 02:44 Copying 2 hours worth to /media/henderson: (later deleted from DRSU) $ ./mesix DR3 CPY "055982_000027273 0 568889893192 /dev/sdg1 120225_0500_B2_NS.dat" UT 120301 02:47 Copying 2 hours worth to /media/anders: (later deleted from DRSU) $ ./mesix DR4 CPY "055982_000027296 0 568889893192 /dev/sdd1 120225_0500_B3_SP.dat" UT 120301 02:55 Deleting the following from DR2 DRSU (already on USB HDD): 055978_000013463 055978 061260000 055978 061870000 DRX_FILT_7 000048197623968 000048681189376 YES (LH001_B2_1) -- /media/borman/LH001_B1_1.dat 055979_000014889 055979 019800000 055979 037810000 DRX_FILT_7 001422938417472 001437274210304 YES (COM02_B2_1 (VirA)) -- /media/Boykin/COM02_0002.dat 055979_000016611 055979 039780000 055979 042780000 DRX_FILT_7 000237024723840 000239413231616 YES (LH002_B2_2) -- /media/borman/LH002_0002.dat 055984_000034134 055984 013800000 055984 014410000 DRX_FILT_7 000048196047072 000048681189376 YES (120227_0350_B2_NP) -- /media/Boykin/055984_000034134_4_DRX.dat UT 120301 03:02 Deleting the following from DR2 DRSU (beam-dipole runs with corrupted beam) 055968_000003638 055968 060900000 055968 068100000 DRX_FILT_7 000568889739936 000574590812160 YES 055968_000003668 055968 068940000 055968 076140000 DRX_FILT_7 000568889752320 000574590812160 YES UT 120301 03:27 Checked station health by doing subset of start-up script UT 120301 03:27 Noticed PASI stopped updating at 2012-02-28 08:14:14 UTC, sent email to staff email exploder UT 120301 03:34 Noticed PASI is updating again -- apparently Hartman intervened UT 120301 03:45 Started 23-hour TBN run at 74.03 MHz (DR says 24 hour run wouldn't fit) ./mesix DR5 REC "55987 13600000 82800000 DEFAULT_TBN" # 23h 055987_000496599 UT 120301 11:17 Copying to /media/boozer: (later deleted from DRSU; data shuffling) $ ./mesix DR4 CPY "055977_000004227 0 1138554710016 /dev/sdb1 LE002_Block04_Sess14_120220.dat" UT 120301 11:40 Deleting the following from DR3 DRSU (Beam-Dipole runs where beam is corrupted): 055968_000000277 055968 008520000 055968 015720000 DRX_FILT_7 000568889537664 000574590812160 YES **CANDIDATE FOR DELETION** 055969_000000153 055969 057840000 055969 065040000 DRX_FILT_7 000568889541792 000574590812160 YES **CANDIDATE FOR DELETION** 055970_000000319 055970 001200000 055970 004800000 DRX_FILT_7 000284444760576 000287295668224 YES **CANDIDATE FOR DELETION** 055973_000003562 055973 006600000 055973 024600000 DRX_FILT_7 001422225251808 001436475981824 YES **CANDIDATE FOR DELETION** UT 120301 12:30: PASI images have been looking bad for about half an hour. Fiddling with the TBN gain: UT 120301 12:44: ./mesix DP_ TBN "74030000 7 19 0" # TBN gain 19 (down 1). Made it worse. UT 120301 12:47: ./mesix DP_ TBN "74030000 7 20 0" # TBN gain 20 (as it was at the start). Back to earlier (bad looking) images UT 120301 12:49: ./mesix DP_ TBN "74030000 7 21 0" # TBN gain 21. Now looks *much* cleaner. UT 120301 14:52 Moved .tgz's and .sdf's to directory "120226_done" in tp directory UT 120301 20:14 Recompiled tpss.c to increase number of observations from 50 to 120; seems to work OK UT 120302 01:05 ./mesix DP_ TBN "74030000 7 20 0" # returned to TBN gain 20; couldn't see Tau A clearly, now I can UT 120302 03:30 Rebooted Exec PC, restarted scheduler and executive UT 120302 04:04 Selected DRSUs: DR1: DRSU02 (UNM), DR2: DRSU02 (UNM), DR3: DRSU02 (UNM), DR4: DRSU01 (UNM) UT 120302 08:43 MGT underflow beam 4 found, did DP INI UT 120302 08:55 LH002 HJ run @ 120302 0849 (4 beams) got started OK with only DRX timeouts: --- BEGIN --- 120302 08:48:54 55988 31734002 T 1024 1 DR4 REC 55988 31740000 10810000 DRX_FILT_7| 120302 08:48:54 55988 31734002 T 1025 1 DR2 REC 55988 31740000 10810000 DRX_FILT_7| 120302 08:48:54 55988 31734002 T 1026 1 DR3 REC 55988 31740000 10810000 DRX_FILT_7| 120302 08:48:54 55988 31734003 T 1027 1 DR1 REC 55988 31740000 10810000 DRX_FILT_7| 120302 08:48:55 55988 31735000 T 1026 2 DR3 REC 55988 31740000 10810000 DRX_FILT_7| 120302 08:48:55 55988 31735000 T 1027 2 DR1 REC 55988 31740000 10810000 DRX_FILT_7| 120302 08:48:55 55988 31735000 T 1024 2 DR4 REC 55988 31740000 10810000 DRX_FILT_7| 120302 08:48:55 55988 31735000 T 1025 2 DR2 REC 55988 31740000 10810000 DRX_FILT_7| 120302 08:48:57 55988 31737002 T 1028 1 DP_ DRX 04014b89544007000700| 120302 08:48:57 55988 31737002 T 1029 1 DP_ DRX 02014c3ebc2007000700| 120302 08:48:57 55988 31737002 T 1030 1 DP_ DRX 03014c3ebc2007000700| 120302 08:48:57 55988 31737002 T 1031 1 DP_ DRX 01014b89544007000700| 120302 08:48:57 55988 31737004 T 1032 1 DP_ DRX 04024c01b32007000700| 120302 08:48:57 55988 31737004 T 1033 1 DP_ DRX 02024c7bc52007000700| 120302 08:48:57 55988 31737004 T 1034 1 DP_ DRX 03024c7bc52007000700| 120302 08:48:57 55988 31737004 T 1035 1 DP_ DRX 01024c01b32007000700| 120302 08:48:57 55988 31737005 T 1036 1 DP_ BAM 4 740_692_2116.df 111226_XY.gf 0| 120302 08:48:57 55988 31737006 T 1037 1 DP_ BAM 2 740_645_1252.df 111226_XY.gf 0| 120302 08:48:57 55988 31737006 T 1038 1 DP_ BAM 3 740_692_2116.df 111226_XY.gf 0| 120302 08:48:57 55988 31737006 T 1039 1 DP_ BAM 1 740_645_1252.df 111226_XY.gf 0| 120302 08:48:58 55988 31738000 T 1037 2 DP_ BAM 2 740_645_1252.df 111226_XY.gf 0| 120302 08:48:58 55988 31738000 T 1038 2 DP_ BAM 3 740_692_2116.df 111226_XY.gf 0| 120302 08:48:58 55988 31738000 T 1039 2 DP_ BAM 1 740_645_1252.df 111226_XY.gf 0| 120302 08:48:58 55988 31738000 T 1028 2 DP_ DRX 04014b89544007000700| 120302 08:48:58 55988 31738000 T 1029 2 DP_ DRX 02014c3ebc2007000700| 120302 08:48:58 55988 31738000 T 1030 2 DP_ DRX 03014c3ebc2007000700| 120302 08:48:58 55988 31738000 T 1031 2 DP_ DRX 01014b89544007000700| 120302 08:48:58 55988 31738000 T 1032 2 DP_ DRX 04024c01b32007000700| 120302 08:48:58 55988 31738000 T 1033 2 DP_ DRX 02024c7bc52007000700| 120302 08:48:58 55988 31738000 T 1034 2 DP_ DRX 03024c7bc52007000700| 120302 08:48:58 55988 31738000 T 1035 2 DP_ DRX 01024c01b32007000700| 120302 08:48:58 55988 31738000 T 1036 2 DP_ BAM 4 740_692_2116.df 111226_XY.gf 0| 120302 08:48:59 55988 31739801 T 1024 3 DR4 REC 055988_000001024| 120302 08:48:59 55988 31739801 T 1025 3 DR2 REC 055988_000001025| 120302 08:48:59 55988 31739801 T 1027 3 DR1 REC 055988_000001027| 120302 08:48:59 55988 31739802 T 1026 3 DR3 REC 055988_000001026| 120302 08:48:59 55988 31739802 T 1037 3 DP_ BAM | 120302 08:49:04 55988 31744810 T 1038 3 DP_ BAM | 120302 08:49:09 55988 31749814 T 1039 3 DP_ BAM | 120302 08:49:14 55988 31754821 T 1028 3 DP_ DRX | 120302 08:49:14 55988 31754821 T 1029 8 DP_ SHT Timed out at subsystem| 120302 08:49:14 55988 31754821 N Previous message: eMIBerror=64 120302 08:49:14 55988 31754821 T 1030 8 DP_ SHT Timed out at subsystem| 120302 08:49:14 55988 31754821 N Previous message: eMIBerror=64 120302 08:49:14 55988 31754821 T 1031 8 DP_ SHT Timed out at subsystem| 120302 08:49:14 55988 31754821 N Previous message: eMIBerror=64 120302 08:49:14 55988 31754821 T 1032 8 DP_ SHT Timed out at subsystem| 120302 08:49:14 55988 31754821 N Previous message: eMIBerror=64 120302 08:49:14 55988 31754821 T 1033 8 DP_ SHT Timed out at subsystem| 120302 08:49:14 55988 31754821 N Previous message: eMIBerror=64 120302 08:49:14 55988 31754822 T 1034 8 DP_ SHT Timed out at subsystem| 120302 08:49:14 55988 31754822 N Previous message: eMIBerror=64 120302 08:49:14 55988 31754822 T 1035 8 DP_ SHT Timed out at subsystem| 120302 08:49:14 55988 31754822 N Previous message: eMIBerror=64 120302 08:49:14 55988 31754822 T 1036 8 DP_ SHT Timed out at subsystem| 120302 08:49:14 55988 31754822 N Previous message: eMIBerror=64 120302 08:49:14 55988 31754830 T 1029 3 DP_ DRX | 120302 08:49:14 55988 31754830 N Previous message: eMIBerror=12 120302 08:49:14 55988 31754832 T 1030 3 DP_ DRX | 120302 08:49:14 55988 31754832 N Previous message: eMIBerror=4 120302 08:49:14 55988 31754834 T 1031 3 DP_ DRX | 120302 08:49:14 55988 31754834 N Previous message: eMIBerror=4 120302 08:49:14 55988 31754835 T 1032 3 DP_ DRX | 120302 08:49:14 55988 31754835 N Previous message: eMIBerror=4 120302 08:49:14 55988 31754837 T 1033 3 DP_ DRX | 120302 08:49:14 55988 31754837 N Previous message: eMIBerror=4 120302 08:49:14 55988 31754839 T 1034 3 DP_ DRX | 120302 08:49:14 55988 31754839 N Previous message: eMIBerror=4 120302 08:49:14 55988 31754840 T 1035 3 DP_ DRX | 120302 08:49:14 55988 31754841 N Previous message: eMIBerror=4 120302 08:49:17 55988 31757001 T 1040 1 DP_ BAM 4 740_692_2118.df 111226_XY.gf 0| 120302 08:49:17 55988 31757001 T 1041 1 DP_ BAM 3 740_692_2118.df 111226_XY.gf 0| 120302 08:49:18 55988 31758000 T 1040 2 DP_ BAM 4 740_692_2118.df 111226_XY.gf 0| 120302 08:49:18 55988 31758000 T 1041 2 DP_ BAM 3 740_692_2118.df 111226_XY.gf 0| 120302 08:49:19 55988 31759839 T 1036 3 DP_ BAM | 120302 08:49:19 55988 31759839 N Previous message: eMIBerror=12 --- END --- UT 120302 12:15 DRSU Utilization (DR1/DRSU2, DR2/DRSU2, DR3/DRSU2, DR4/DRSU1): -- these are UNM DRSUs Was (%) 100.0 98.9 74.8 72.9 before above observation Now (%) 91.3 90.3 66.2 64.3 after above observation (so, each recording about 8.7%) UT 120302 12:35 Noticed PASI (LWA-TV) has been stalled since 08:40. Sent email to staff email exploder. UT 120302 12:36 Sent DP_ INI to turn off beams and reset TBN. First & second attempts failed; third attempt worked. UT 120302 12:42 Restarted TBN with ./mesix DP_ TBN "74030000 7 20 0" UT 120302 12:45 SELected VT DRSUs: DR1: DRSU01, DR2: DRSU01, DR3: DRSU01, DR4: DRSU02 UT 120302 13:24 MGT underflow -- beam 3 cal lost. Doing DP_ INI. Worked first time. UT 120302 13:27 Restarted TBN with ./mesix DP_ TBN "74030000 7 20 0" UT 120302 13:32 MGT underflow -- beam 3 cal lost. Doing DP_ INI. First/second/third attempts failed; fourth attempt succeeded UT 120302 13:48 Restarted TBN with ./mesix DP_ TBN "74030000 7 20 0" UT 120302 13:57 Sent SYN to all 5 DRs; in each case I got acceptance with no data returned to indicate correct time UT 120302 13:57 MGT underflow -- beam 3 cal lost. Again. Letting it ride this time. UT 120302 13:57 SHL temp is 63.86 UT 120302 14:02 SHL temp is 66.56 UT 120302 14:07 SHL temp is 69.26 UT 120302 14:12 SHL temp is 65.48; OK, so temperature is just jumping around UT 120302 14:14 Doing DP_ INI. Want to see if beam3 cal is related turning TBN on. INI worked first time. UT 120302 14:17 SHL temp is 60.08 UT 120302 14:22 SHL temp is 56.84 UT 120302 14:39 Commissioning experiment: Beam 1 fixed at location of Cyg A transit, Beam 2 fixed at location of Cas A transit Starting 1 hour prior to Cyg A transit, ending 1 hour after Cas A transit In both cases, using x-pol of beam and x-pol of outrigger (258) --- astronomical details: --- Transit Transit-1h Transit-1h RA [h] DEC [d] MT ET UT MT ET UT MPM Date MJD Cyg A 19.991 +40.734 09:31 11:31 16:31 08:31 10:31 15:31 55860000 # 120302 Transits at Alt=83:19=83.317 Az=0 Transit+1h Cas A 23.391 +58.800 12:54 14:54 19:54 13:54 15:54 20:54 # 120302 Transits at Alt=65:13=65.217 Az=0 Dur: 5:23 = 19380000 ms --- set up --- ./mefsdfg state dfiles 74 1 83.317 0 (already done) ./mefsdfg state dfiles 74 1 65.217 0 (already done) The .df's were transferred to sch/dfiles --- run ------ The following commands given around time indicated by this log entry: ./mesix DP_ DRX "1 1 39400000 7 7 0" ./mesix DP_ DRX "1 2 73700000 7 7 0" ./mesix DP_ BAM "1 740_833_0000.df 111226_XO.gf 0" # Beam fixed at Cyg A transit ./mesix DP_ DRX "2 1 39400000 7 7 0" ./mesix DP_ DRX "2 2 73700000 7 7 0" ./mesix DP_ BAM "2 740_652_0000.df 111226_XO.gf 0" # Beam fixed at Cas A transit --- recording details --- ./mesix DR1 REC "55988 55860000 19380000 DRX_FILT_7" # Start UT 120302 15:31, Dur 5:23, expecting 1426.1 GB 120302 15:00:17 55988 54017832 T 3998 3 DR1 REC 055988_000003998| ./mesix DR2 REC "55988 55860000 19380000 DRX_FILT_7" # Start UT 120302 15:31, Dur 5:23, expecting 1426.1 GB 120302 15:00:30 55988 54030250 T 3999 3 DR2 REC 055988_000003999| --- end ---- UT 120302 15:10 Checked time on DRs 1-4. DRs 1-3 were exact as far as I can tell. DR4 was 8 seconds behind; manually corrected using "date" command UT 120302 15:31 COM run (Cas/CygA fixed transit beams) (Beams 1&2) -- done 2053 UT [done 1553 ET] UT 120302 17:41 COMM_022 (Cas/CygA/NCP rotation) started on Beam 4. Completed successfully, no issues. UT 120303 01:01 Started CGP run LE002_120303_0101_0026_B_GP through 27,28,29 (4 beams). No dropped or late DP commands! However, DR1 and DR4 could only accomodate tiny files -- evidentally due to DRSU fragmentation UT 120303 04:01 Switching to UNM DRSUs (DR1/DRSU2, DR2/DRSU2, DR3/DRSU2, DR4/DRSU1) UT 120303 04:10 Checked time on DRs 1-4. DRs 1-3 were exact as far as I can tell. DR4 was 4 seconds behind; manually corrected using "date" command UT 120303 08:45 Started HJ run LH002_120303_0845_00##_B* (sessions 35-38) (4 beams). No dropped or late DP commands! However, DR4 could only accomodate 1.39 h of requested 4 h -- evidentally due to DRSU fragmentation UT 120303 12:10-ish DP_ INI to clear "Warning condition cleared" message in INFO field (due to temp warnings overnight) UT 120303 12:14 $ ./mesix DP_ TBN "74030000 7 20 0" to restart TBN/PASI UT 120303 12:44 Sag A drift 1244-1544 SPC Beam 1 @ 26.917 deg alt $ ./mefsdfg state dfiles 74 1 26.917 180 ---> 740_269_1800.df copied to sch/dfiles ./mesix DP_ DRX "1 1 37900000 1 7 0" # 250 kSPS ./mesix DP_ DRX "1 2 74030000 1 7 0" # 250 kSPS ./mesix DP_ BAM "1 740_269_1800.df 111226_XO.gf 0" # Beam fixed at Sag A transit ./mesix DR1 SPC "55989 45840000 10800000 32 6144" # Start UT 120303 12:44, Dur 3h, expecting 7.37 MB tag 055989_00011670 UT 120303 14:45 DP_ INI to clear MGT underflow on beam 3; reapplied DP beam 1 settings for above observation. UT 120303 14:50 Discovered DR1 is not responding -- apparently something in DP INI / beam reset borked it. Emailed Chris for suggestion. UT 120303 14:57 Cyg A drift 1457-1757 SPC Beam 2 @ 83.3 deg alt ./mesix DP_ DRX "2 1 37900000 1 7 0" # 250 kSPS ./mesix DP_ DRX "2 2 74030000 1 7 0" # 250 kSPS ./mesix DP_ BAM "2 740_833_0000.df 111226_XO.gf 0" # Beam fixed at Cyg A transit ./mesix DR2 SPC "55989 53820000 10800000 32 6144" # Start UT 120303 14:57, Dur 3h, expecting 7.37 MB tag 055989_00012230 UT 120303 17:53 Sun drift 1753-2053 SPC Beam 3 @ 49.4 deg alt $ ./mefsdfg state dfiles 74 1 49.483 180 ---> 740_269_1800.df copied to sch/dfiles ./mesix DP_ DRX "3 1 37900000 1 7 0" # 250 kSPS ./mesix DP_ DRX "3 2 74030000 1 7 0" # 250 kSPS ./mesix DP_ BAM "3 740_494_1800.df 111226_XO.gf 0" # Beam fixed at Sun transit ./mesix DR3 SPC "55989 64380000 10800000 32 6144" # Start UT 120303 17:53, Dur 3h, expecting 7.37 MB tag 12321 UT 120303 18:20 Cas A drift 1820-2120 SPC Beam 4 @ 65.2 deg alt ./mesix DP_ DRX "4 1 37900000 1 7 0" # 250 kSPS ./mesix DP_ DRX "4 2 74030000 1 7 0" # 250 kSPS ./mesix DP_ BAM "4 740_652_0000.df 111226_XO.gf 0" # Beam fixed at Cas A transit ./mesix DR4 SPC "55989 66000000 10800000 32 6144" # Start UT 120303 18:20, Dur 3h, expecting 7.37 MB tag 12342 UT 120303 21:32 Restarted TBN: ./mesix DP_ TBN "74030000 7 20 0" UT 120303 22:03 Per suggestion from Chris, was able to revive DR1 as follows: joecraig@DR:~$ ps -e | grep WAVE_Y # confirming WAVE_Y is dead... should be joecraig@DR:~$ killall WAVE_Y # making sure WAVE_Y is dead WAVE_Y: no process found joecraig@DR:~$ sudo service StartDROS.sh start & then exit; should start working after a couple minutes UT 120303 22:03 Selected DRSU #1 on DR2 and DRSU#2 on DR3 (VT DRSUs) in prep for CGP run UT 120303 22:03 Checked times on DR#2 and #3: They are essentially exact WRT MCS/TP. UT 120303 22:33 Did a DP_ INI to shutdown TBN and "refresh" DP generally. Worked first time. UT 120304 00:57 CGP run. No timeouts or DP dropped commands! LE002_120304_0057_0030_B_GP.sdf -- 1hr on beam 2 starting 00:57 -- DRSU LE002_120304_0057_0031_B_NS.sdf -- 1hr on beam 3 starting 00:57 -- DRSU UT 120304 02:00 CGP run. All SPC-mode. No timeouts or DP dropped commands! LE002_120304_0200_0032_S_GP.sdf -- 1hr on beam 1 starting 02:00 -- SPC LE002_120304_0200_0033_S_GP.sdf -- 1hr on beam 2 starting 02:00 -- SPC LE002_120304_0200_0034_S_NS.sdf -- 1hr on beam 3 starting 02:00 -- SPC LE002_120304_0200_0035_S_NS.sdf -- 1hr on beam 4 starting 02:00 -- SPC UT 120304 03:32 Switched to UNM DRSUs UT 120304 03:37 Checked DRs 1-4 for time sync. DRs 1-3 were exact best I could tell. DR4 was 6s slow. Manually corrected using "date" command UT 120304 08:41 HJ (Tau Boo) run. No timeouts or DP dropped commands! LH002_120304_0841_0039_B_HJ.sdf -- 3hr on beam 1 --- DRSU tag 055990_000016824 LH002_120304_0841_0040_B_HJ.sdf -- 3hr on beam 2 --- DRSU tag 055990_000016825 LH002_120304_0841_0041_B_NS.sdf -- 3hr on beam 3 --- DRSU tag 055990_000016826 LH002_120304_0841_0042_S_NS.sdf -- 3hr on beam 4 --- SPC (due to lack of space) tag 055990_000016827 UT 120304 11:43 Switched to VT DRSUs UT 120304 11:49 Checked DRs 1-4 for time sync. DRs 1-3 were exact best I could tell. DR4 was 1s slow. Made not attempt to correct. UT 120304 13:36 SDP run towards GCRT (DR2/DRSU, DR1/SPC elsewhere for anticoincidence) 120304 1336-1436 [ET: 0836-0936] LS001_120304_1336_0026_B_SP.sdf -- 1hr on beam 2 (@GC) starting 13:36 -- DRSU tag 055990_000019479 LS001_120304_1336_0027_S_SP.sdf -- 1hr on beam 3 (@RF) starting 13:36 -- SPC tag 055990_000019480 First DRX command for each beam timed out, but eventually answered. No commands dropped. UT 120304 14:42 B1919+21 in SPC mode 1442-1642 [ET 0944-1144] No timeouts or DP dropped commands! COMM_120304_1442_0023_S_NS.sdf -- 1hr on beam 1 (74 & 59 MHz) -- SPC tag 055990_000020087 COMM_120304_1442_0024_S_NS.sdf -- 1hr on beam 2 (38 & 20 MHz) -- SPC tag 055990_000020088 UT 120304 16:53 Restarted MCS/Exec to abort Sun tracking observation. Done with observing for a while. UT 120304 16:54 Restarted TBN: ./mesix DP_ TBN "74030000 7 20 0" -- looks OK UT 120304 19:21 Retuned TBN: ./mesix DP_ TBN "25610000 7 20 0" -- Strong narrowband RFI evident Retuned TBN to 25.71 MHz -- PASI stalled. Sent email to LWA1 staff exploder UT 120304 19:38 Tried a DP_ INI followed by a TBN command. PASI still stalled; suggests PASI is the problem UT 120304 19:53 DP now indicating an MGT underflow on beam 3 -- circumstantial evidence points to TBN UT 120304 21:28 Rebooted MCS/TP PC. (no particular reason -- just wanted a clean slate and check station start-up instructions) UT 120304 21:51 Switched to UNM DRSUs on DR1 and DR2. UT 120305 21:53 Checked sync on DR1 and DR2 -- they seem to be synced with MCS/TP. MJD = 55991 UT 120306 00:33 Did a DP_ INI to turn off TBN. First attempt failed, second succeeded UT 120305 01:03 Jupiter Mar 05 01:03 [ET Mar 04 20:03-22:34]. No dropped commands or timeouts at startup! LC001_120305_0103_0016_B_JP.sdf -- 2h31min on beam 1 (Jupiter, 20 & 26 MHz) -- DRSU tag ends in 983 LC001_120305_0103_0017_B_M1.sdf -- 2h31min on beam 1 (M1, 20 & 26 MHz) -- DRSU tag ends in 984 UT 120305 03:48 Checked sync on DRs -- all but DR4 seem to be synced with MCS/TP; DR4 about 5 s off; fixed manually UT 120305 10:38 Lost all communication with site. Can't tell if a power failure or a network outage. Sent email to LWA1 staff email exploder. UT 120305 13:16 Still unable to communicate with site. --- completed, but not yet logged -------------------------------------------------- --- running ------------------------------------------------------------------------ --- entered into MCS --------------------------------------------------------------- UT 120305 08:37 HJ (Tau Boo) run. LH002_120305_0837_0043_B_HJ.sdf -- 3hr on beam 1 --- DRSU tag LH002_120305_0837_0044_B_HJ.sdf -- 3hr on beam 2 --- DRSU tag LH002_120305_0837_0045_B_NS.sdf -- 3hr on beam 3 --- DRSU tag LH002_120305_0837_0046_S_NS.sdf -- 3hr on beam 4 --- SPC (due to lack of space) tag --- planned ---------------------------------------------------------------- =================================================================== issues: UT 120229 17:00 "$ shutdown -g0" strange for DR2-5. Get shutdown message, get kicked out, can't log back in, but still responding to "$ ping" ------------------------------------------------------------------- Ext HDD Utilization DR1 joecraig@DR:~$ df -H Filesystem Size Used Avail Use% Mounted on /dev/sdf1 2.0T 642G 1.3T 35% /media/haise -rw-r--r-- 1 root root 640790959104 2012-02-23 06:25 LC001_B1_1.dat -rw-r--r-- 1 joecraig joecraig 5258129 2012-03-03 08:08 055989_000011670_spectrometer.dat -rw-r--r-- 1 joecraig joecraig 51193499 2012-03-03 20:26 055990_000014872_spectrometer.dat -rw-r--r-- 1 joecraig joecraig 408980130 2012-03-04 11:09 055990_000020087_spectrometer.dat /dev/sdg1 2.0T 1.2T 731G 61% /media/caldbeck -rw-r--r-- 1 root root 568889893192 2012-02-29 17:35 120224_0500_B1_NS.dat -rw-r--r-- 1 root root 568889893192 2012-03-01 00:37 120225_0500_B1_NS.dat /dev/sdh1 2.0T 71M 1.9T 1% /media/swigert DR2 mcsdr@mcs-dr2:~$ df -H Filesystem Size Used Avail Use% Mounted on /dev/sdh1 2.0T 1.5T 397G 79% /media/Boykin -rw-r--r-- 1 root root 1422938417472 2012-02-24 05:10 COM02_0002.dat -rw-r--r-- 1 root root 48196047072 2012-02-26 21:29 055984_000034134_4_DRX.dat /dev/sdf1 2.0T 387G 1.5T 21% /media/borman -rw-r--r-- 1 root root 100328980128 2012-02-17 14:44 120217_1712_B2_CygA.dat -rwxr-xr-x 1 mcsdr mcsdr 13351 2012-02-17 15:18 drxWaterfallSkip.py -rw-r--r-- 1 mcsdr mcsdr 157357348 2012-02-17 16:45 120217_1712_B2_CygA-waterfall-full.npz -rw-r--r-- 1 mcsdr mcsdr 39414148 2012-02-17 17:29 120217_1712_B2_CygA-waterfall.npz -rw-r--r-- 1 root root 237024723840 2012-02-24 14:54 LH002_0002.dat -rw-r--r-- 1 root root 48197623968 2012-02-24 15:35 LH001_B2_1.dat -rw-r--r-- 1 mcsdr mcsdr 7813508 2012-03-03 12:34 055989_000012230_spectrometer.dat -rw-r--r-- 1 mcsdr mcsdr 51192361 2012-03-03 20:22 055990_000014873_spectrometer.dat -rw-r--r-- 1 mcsdr mcsdr 408977285 2012-03-04 11:11 055990_000020088_spectrometer.dat /dev/sdg1 2.0T 254G 1.7T 14% /media/roosa total 0 DR3 mcsdr@mcs-dr3:~$ df -H Filesystem Size Used Avail Use% Mounted on /dev/sdf1 2.0T 642G 1.3T 35% /media/scott__ -rw-r--r-- 1 root root 640791706272 2012-02-22 16:52 LC001_B3_1.dat -rw-r--r-- 1 mcsdr mcsdr 7813508 2012-03-03 14:22 055989_000012321_spectrometer.dat -rw-r--r-- 1 mcsdr mcsdr 51193499 2012-03-03 20:19 055990_000014874_spectrometer.dat -rw-r--r-- 1 mcsdr mcsdr 204772858 2012-03-04 11:04 055990_000019480_spectrometer.dat /dev/sdg1 2.0T 1.2T 731G 61% /media/henderson -rw-r--r-- 1 root root 568889893192 2012-02-29 17:44 120224_0500_B2_NS.dat -rw-r--r-- 1 root root 568889893192 2012-03-01 00:39 120225_0500_B2_NS.dat DR4 mcsdr@mcs-dr4:~$ df -H Filesystem Size Used Avail Use% Mounted on /dev/sdd1 2.0T 1.5T 426G 78% /media/anders -rw-r--r-- 1 root root 568889893192 2012-02-29 17:59 120224_0500_B3_SP.dat -rw-r--r-- 1 root root 568889893192 2012-03-01 00:41 120225_0500_B3_SP.dat /dev/sdb1 2.0T 1.2T 730G 61% /media/boozer -rw-r--r-- 1 root root 1138554710016 2012-03-01 14:08 LE002_Block04_Sess14_120220.dat /dev/sdc1 2.0T 25G 1.9T 2% /media/duke -rw-r--r-- 1 root root 24493669632 2012-02-18 17:46 055976_000000084_4_DRX.dat /dev/sda1 2.0T 955G 915G 52% /media/dunleavy -rw-r--r-- 1 root root 284415133920 2012-02-11 23:52 120210_2000_MT.dat -rw-r--r-- 1 root root 100329661248 2012-02-15 11:48 120215_1720_CygA.dat -rw-r--r-- 1 root root 568885067040 2012-02-22 16:34 LC001_B4_1.dat -rw-r--r-- 1 mcsdr mcsdr 7814077 2012-03-03 14:36 055989_000012342_spectrometer.dat -rw-r--r-- 1 mcsdr mcsdr 51196344 2012-03-03 20:17 055990_000014875_spectrometer.dat -rw-r--r-- 1 mcsdr mcsdr 613228939 2012-03-04 10:53 055990_000016827_spectrometer.dat DR5 mcsdr@comm-mcs-dr5:~$ df -H Filesystem Size Used Avail Use% Mounted on /dev/sdi1 2.0T 71M 1.9T 1% /media/lovell /dev/sdf1 2.0T 332G 1.6T 18% /media/young -rw-r--r-- 1 root root 296063329496 2012-02-20 13:10 120213_0020_TN_26.dat -rw-r--r-- 1 root root 28246827840 2012-02-22 11:25 TBN_bad_timetags.dat -rw-r--r-- 1 root root 6386106424 2012-02-27 14:36 TBN_single_BAM_timetags.dat /dev/sdh1 2.0T 71M 1.9T 1% /media/worden =============================== DRSU Utilization TAG--------------StaMJD-StaMPM----StpMJD-StpMPM--------------------------Format-----Size------------Alloc-----------Cp? DR1 DRSU01: /dev/md0 [VT DRSU] 67.8% remaining -- maxed out 055980_000018158 055980 003120000 055980 017530000 DRX_FILT_7 001138493698176 001149979066368 YES (LE002_Block05_Sess17_120223) 055983_000031774 055983 002040000 055983 008450000 DRX_FILT_7 000506434784160 000511545442304 YES (LE002_Block06_Sess22_120226) 055988_000003998 055988 055860000 055988 075240000 DRX_FILT_7 001531262049792 001546605821952 YES (Cyg A beam-dipole 120302) 055989_000006530 055989 004000000 055989 004100000 DRX_FILT_7 000007901256192 000007980974080 YES (LE002_120303_0101_0026_B_GP.sdf) DRSU02: /dev/md1 66.8% remaining 055988_000001027 055988 031740000 055988 042550000 DRX_FILT_7 000854124942336 000862683922432 YES (LH002_0031) 055989_000009300 055989 031500000 055989 042310000 DRX_FILT_7 000854125119840 000862683922432 YES (LH002_0035) 055990_000016824 055990 031260000 055990 042070000 DRX_FILT_7 000854127786528 000862683922432 YES (LH002_0039) 055991_000000983 055991 003780000 055991 012850000 DRX_FILT_7 000716645782656 000723824672768 YES (LC001_0016) DR2 1: DRSU01: /dev/md0 [VT DRSU] 61.5% remaining 055983_000031728 055983 001440000 055983 015850000 DRX_FILT_7 001118824042368 001149979066368 YES (LE002_Block06_Sess23_120226, 19.8/59.0) 055988_000003999 055988 055860000 055988 075240000 DRX_FILT_7 001531261884672 001546605821952 YES (Cas A beam-dipole 120302) 055989_000006455 055989 003660000 055989 010870000 DRX_FILT_7 000569679954720 000575388778496 YES (LE002_120303_0101_0027_B_GP.sdf) 055990_000014320 055990 003420000 055990 007030000 DRX_FILT_7 000285234781344 000288093896704 YES (LE002_120304_0057_0030_B_GP.sdf) 055990_000019479 055990 048960000 055990 052570000 DRX_FILT_7 000285234789600 000288093896704 YES (LS001_120304_1336_0026_B_SP.sdf) 2: DRSU02: /dev/md1 65.8% remaining 055974_000000105 055974 060672000 055974 061942000 DRX_FILT_7 000100328980128 000101351948288 YES (Not in log) 055988_000001025 055988 031740000 055988 042550000 DRX_FILT_7 000854124929952 000862683922432 YES (LH002_0032) 055989_000009301 055989 031500000 055989 042310000 DRX_FILT_7 000854124446976 000862683922432 YES (LH002_0036) 055990_000016825 055990 031260000 055990 042070000 DRX_FILT_7 000854124517152 000862683922432 YES (LH002_0040) 055991_000000984 055991 003780000 055991 012850000 DRX_FILT_7 000716643520512 000723824672768 YES (LC001_0017) DR3 1: DRSU01: /dev/md0 [VT DRSU] 88.7% remaining -- maxed out 055983_000031779 055983 002040000 055983 008450000 DRX_FILT_7 000506470264320 000511545442304 YES (LE002_Block06_Sess24_120226) 055989_000006456 055989 003660000 055989 010870000 DRX_FILT_7 000569679987744 000575388778496 YES (LE002_120303_0101_0028_B_NS.sdf) 055990_000014362 055990 003700000 055990 004200000 DRX_FILT_7 000039506252064 000039902773248 YES (LE002_120304_0057_0031_B_NS.sdf) 2: DRSU02: /dev/md1 48.9% remaining 055968_000001443 055968 024240000 055968 031440000 DRX_FILT_7 000568889512896 000574590812160 YES (DD) 055968_000002200 055968 036180000 055968 043380000 DRX_FILT_7 000568889504640 000574590812160 YES (DD) 055979_000015530 055979 038700000 055979 045910000 DRX_FILT_7 000569680214784 000575388778496 YES (LH002_B3_1) 055982_000024829 055982 002100000 055982 012010000 DRX_FILT_7 000783013554048 000790860136448 YES (LC001_B2_2 (NCP)) 055988_000001026 055988 031740000 055988 042550000 DRX_FILT_7 000854125107456 000862683922432 YES (LH002_0033) 055989_000009302 055989 031500000 055989 042310000 DRX_FILT_7 000854124797856 000862683922432 YES (LH002_0037) 055990_000016826 055990 031260000 055990 042070000 DRX_FILT_7 000854125107456 000862683922432 YES (LH002_0041) DR4 1: DRSU01: /dev/md0 60.3% remaining -- maxed out 055976_000000084 055976 001500000 055976 001810000 DRX_FILT_7 000024493669632 000024739840000 YES (120219_2230_B4_NP) 055976_000000269 055976 003180000 055976 017590000 DRX_FILT_7 001138560270432 001149979066368 YES (LE002_Block03_Sess12_120210) 055979_000014599 055979 004500000 055979 011700000 DRX_FILT_7 000568885067040 000574590812160 YES (LC001_B4_1 (CasA)) 055979_000016490 055979 039600000 055979 045900000 DRX_FILT_7 000497774446560 000502767026176 YES (LH002_B4_1 (Ref)) 055982_000024873 055982 002700000 055982 008410000 DRX_FILT_7 000451157483328 000455682555904 YES (LC001_B2_2 (CasA)) 055988_000001024 055988 031740000 055988 042550000 DRX_FILT_7 000854184620832 000862683922432 YES (LH002_0034) 055989_000009393 055989 031900000 055989 036900000 DRX_FILT_7 000395090037888 000399021703168 YES (LH002_0038) 2: DRSU02: /dev/md1 [VT DRSU] 88.4% remaining -- maxed out 055983_000031782 055983 002040000 055983 008450000 DRX_FILT_7 000506465719392 000511545442304 YES (LE002_Block06_Sess25_120226) 055988_000004544 055988 063660000 055988 070870000 DRX_FILT_7 000569617027488 000575388778496 YES (COMM_0022. CasA/CygA/NCP rotation) 055989_000006555 055989 004100000 055989 004900000 DRX_FILT_7 000063214408704 000063844122624 YES (LE002_120303_0101_0029_B_NS.sdf) DR5 1: DRSU01: /dev/md0 1.8% remaining 055987_000496599 055987 013600000 055988 010000000 DEFAULT_TBN 008810301719600 009821315923968 YES