Started Cyg A/Cas A beam + outlier drift data collection. From Steve: "Here's a commissioning run I would like to do. It points beam 1 (fixed) at the position where Cyg A transits, and beam 2 (fixed) at the position where Cas A transits, and lasts 5 hours (during which Cyg A transits through its beam, and later Cas A transits through its beam). For each beam, one pol of the beam is the beam 'x' pol, and the other pol of the beam is the 'x' pol of the outrigger. This is currently not possible to do in MCS0030, but is easy from the command line (script provided below). This serves a couple purposes: Beam shape mapping, far-out sidelobe mapping (by cross correlation among beams and outriggers), and sensitivity calculation (by comparison of in- and out-of-beam levels). " DR1 (Cyg A) was started 2 minutes late (16:32 UT) due to operator error. DR 2 (Cas A) reported insufficient drive space. Switched over to the second DRSU and DROS died. Rebooted DR2 and the recording was accepted (16:48 UT). The DR1 file is strangely small (only ~34 GB). Could be a configuration issue since the North Arm DR was stolen to replace a broken DR1. The DR2 file looks to be the right size. Should redo the Cyg A stuff. Started NPC + DR Spectrometer observation for Chris. From Chris: "Would it be possible at some point to run a short (~5 minutes) SPC recording (at full DRX rate) and grab both it and the log file? I'm getting blank spectra and it looks (on the surface) like DP is sending zero-filled data. Since I don't believe this is actually happening, I'm trying to identify where in my code the data is getting lost, and without a log file corresponding to a SPC recording, I'm a little in the dark." Also captured a simultaneous beam 4 recording with the same bandwidth/tuning for cross checking. CGP Block 3 (LE002, sessions 9, 10, 11, 12): DR1 recording size is unusually small again. Need to check the formats there. Simultaneous TBN not captured. CGP Block 4 (LE002, sessions 13 and 14): Going with only two beams tonight due to the DP thermal issue. Session 13 is on beam 2 and session 14 is on beam 4. TBN tuned to 74.03 MHz. The TBN doesn't seem to have recorded for some reason since the last directory entry (#14) is dated MJD 55972. Pulsars: Ran LR001_1.sdf on beam 2. The first DRX commands seem to have been missed by DP. DRX sent again with gain of 10 at 6:04 UT. INI'd DP before starting this block to clear the MGT underflow on beam 3. Changed DRX gain to 7 for both tunings at 6:09 UT. CygA: Ran a 5 minute capture of Cyg A on beam 1 started at 15:55 UT to test the changes to DR1. The file size reports to be 7028418528 bytes (6.5 GB) whereas I expect about 22 GB. Re-running the Cyg A beam on DR2 for 5 hours starting at 16:30 UT. Ran a 5 minute capture of Cyg A on beam 1 starting at 5:11 UT to test the changes to DR1. Recorded zero size, trying again at 5:25 UT after setting DRX and BAM. That seems to have worked. Started the LH001 collection for Cyg A at 17:01 UT. Running on beams 1 and 2. I switched the DR DRSU selections to make sure that things aren't being recorded to the Long Island DRSUs. Beam 1, tuning 1 missed its DRX, resent at 17:02 UT. DR2 shows no DRSUs after reboot. Very strange. Jupiter: Started the Io-A event Jupiter observations on beams 1 (Jupiter), 3 (NCP), and 4 (Cas A) + TBN. DR4 didn't start due to "invalid time". Lots of those in the mselog.txt. Maybe I should save a copy... There was also a message "DR4 REC Timed out at ms_mcic" . Something's very weird here. DR5 took 5 seconds to respond to a RPT. DR1 should be recording to a LWA DRSU. I'm not so sure about 3 and 4 . Nope, 3 is going to a VT DRSU. I think 4 is ok. DR2 now shows one DRSU, it seems to be a LWA one. Switched DR3 over to the LWA DRSU. I'm not really sure why the DRSUs disappeared from DR2. Deep Integrations: Started a two-part deep integration test. Beam 1 on NCP, beam 2 tracking Vir A as it rises. 5 hours per beam at 38 and 74 MHz. Tau Boo: All beams, 2 and 4 didn't start due to insufficient space and invalid time. SYN'd DR2 and 4. DR4 REC started at 11:00 UT with 1.75 hour recording. DR2 wouldn't start recording until recording time was <1 hour (3k seconds). DR2 REC started at 11:03 UT. TBN Stangeness: Jake reported strange time tags coming out of TBN. From Jake: Rank 0 -- Dig board 0 (chans 000-019): +180186030080 (+919.316 s) Rank 0 -- Dig board 1 (chans 020-039): +180186030080 (+919.316 s) Rank 0 -- Dig board 2 (chans 040-059): +180186030080 (+919.316 s) Rank 0 -- Dig board 3 (chans 060-079): +180186030080 (+919.316 s) Rank 0 -- Dig board 4 (chans 080-099): +180186030080 (+919.316 s) Rank 0 -- Dig board 5 (chans 100-119): +180186030080 (+919.316 s) Rank 0 -- Dig board 6 (chans 120-139): +180186030080 (+919.316 s) Rank 0 -- Dig board 7 (chans 140-159): +180186030080 (+919.316 s) Rank 0 -- Dig board 8 (chans 160-179): +180186030080 (+919.316 s) Rank 0 -- Dig board 9 (chans 180-199): +180186030080 (+919.316 s) Rank 0 -- Dig board 10 (chans 200-219): +180186030080 (+919.316 s) Rank 0 -- Dig board 11 (chans 220-239): +180186030080 (+919.316 s) Rank 0 -- Dig board 12 (chans 240-259): +180186030080 (+919.316 s) Rank 0 -- Dig board 13 (chans 260-279): +180186030080 (+919.316 s) Rank 0 -- Dig board 14 (chans 280-299): +180186030080 (+919.316 s) Rank 0 -- Dig board 15 (chans 300-319): +180186030080 (+919.316 s) Rank 0 -- Dig board 16 (chans 320-339): +180186030080 (+919.316 s) Rank 0 -- Dig board 17 (chans 340-359): +180186030080 (+919.316 s) Rank 0 -- Dig board 18 (chans 360-379): +180186030080 (+919.316 s) Rank 0 -- Dig board 19 (chans 380-399): +169277767680 (+863.662 s) Rank 0 -- Dig board 20 (chans 400-419): +169277767680 (+863.662 s) Rank 0 -- Dig board 21 (chans 420-439): +169277767680 (+863.662 s) Rank 0 -- Dig board 22 (chans 440-459): +169277767680 (+863.662 s) Rank 0 -- Dig board 23 (chans 460-479): +67990487040 (+346.890 s) Rank 0 -- Dig board 24 (chans 480-499): +38314393600 (+195.482 s) Rank 0 -- Dig board 25 (chans 500-519): +0 (+0.000 s) The board time (as seen by 'date') looks fine. I took a 5 minute TBN capture starting at 17:30 UT to get some data about how bad things are. TBN CPU times (last column) are reported by the boards: ops@lwa-dp:/lwa/software$ python boardCommand.py defaults.cfg "ps -eo uid,pid,cmd,time | grep tbn" 300 32603 /lwa/software/tbn 192.168.0 04:30:04 300 32465 /lwa/software/tbn 192.168.0 04:30:04 300 32607 /lwa/software/tbn 192.168.0 04:30:06 300 32601 /lwa/software/tbn 192.168.0 04:30:07 300 32601 /lwa/software/tbn 192.168.0 04:30:12 300 32607 /lwa/software/tbn 192.168.0 04:30:11 300 32605 /lwa/software/tbn 192.168.0 04:30:09 300 32538 /lwa/software/tbn 192.168.0 04:30:12 300 32605 /lwa/software/tbn 192.168.0 04:30:09 300 32457 /lwa/software/tbn 192.168.0 04:30:09 300 32605 /lwa/software/tbn 192.168.0 04:30:14 300 32599 /lwa/software/tbn 192.168.0 04:30:12 300 31929 /lwa/software/tbn 192.168.0 04:30:14 300 32601 /lwa/software/tbn 192.168.0 04:30:14 300 32460 /lwa/software/tbn 192.168.0 04:30:15 300 32620 /lwa/software/tbn 192.168.0 04:30:05 300 32617 /lwa/software/tbn 192.168.0 04:30:11 300 32619 /lwa/software/tbn 192.168.0 04:29:57 300 541 /lwa/software/tbn 192.168.0 04:29:46 300 32615 /lwa/software/tbn 192.168.0 04:29:09 300 32617 /lwa/software/tbn 192.168.0 00:17:05 300 32619 /lwa/software/tbn 192.168.0 00:17:02 300 32471 /lwa/software/tbn 192.168.0 04:27:12 300 32619 /lwa/software/tbn 192.168.0 04:26:27 300 32607 /lwa/software/tbn 192.168.0 04:25:30 300 31929 /lwa/software/tbn 192.168.0 02:35:43 Very strange indeed. After resetting TBN and waiting a bit: ops@lwa-dp:/lwa/software$ python boardCommand.py defaults.cfg "ps -eo uid,pid,cmd,time | grep tbn" 300 11448 /lwa/software/tbn 192.168.0 00:15:41 300 11320 /lwa/software/tbn 192.168.0 00:15:41 300 11452 /lwa/software/tbn 192.168.0 00:15:41 300 11454 /lwa/software/tbn 192.168.0 00:15:41 300 11446 /lwa/software/tbn 192.168.0 00:15:41 300 11452 /lwa/software/tbn 192.168.0 00:15:41 300 11450 /lwa/software/tbn 192.168.0 00:15:41 300 11383 /lwa/software/tbn 192.168.0 00:15:41 300 11450 /lwa/software/tbn 192.168.0 00:15:41 300 11302 /lwa/software/tbn 192.168.0 00:15:41 300 11450 /lwa/software/tbn 192.168.0 00:15:41 300 11444 /lwa/software/tbn 192.168.0 00:15:41 300 10774 /lwa/software/tbn 192.168.0 00:15:42 300 11446 /lwa/software/tbn 192.168.0 00:15:42 300 11304 /lwa/software/tbn 192.168.0 00:15:42 300 11492 /lwa/software/tbn 192.168.0 00:15:41 300 11470 /lwa/software/tbn 192.168.0 00:15:42 300 11464 /lwa/software/tbn 192.168.0 00:15:42 300 11846 /lwa/software/tbn 192.168.0 00:15:42 300 11460 /lwa/software/tbn 192.168.0 00:15:42 300 11462 /lwa/software/tbn 192.168.0 00:15:42 300 11464 /lwa/software/tbn 192.168.0 00:15:41 300 11316 /lwa/software/tbn 192.168.0 00:15:42 300 11464 /lwa/software/tbn 192.168.0 00:15:42 300 11466 /lwa/software/tbn 192.168.0 00:15:42 300 10784 /lwa/software/tbn 192.168.0 00:15:42 CGPs: LE002, block 5 with CGP on beams 1 and 2, blank field on beams 3 and 4, and TBN running. REC rejected on beams 1, 2, and 4. MGT underflow on beam 3 ~4 minutes in. Multiple BAMs missed. DR1 still won't record with "operation not permitted." Fixed this by taking down DROS on DR1 and formatting the VT DRSU. Recording on beam 1 started at 0:52 UT. I was able to start DR4 recording (invalid time?) and send all of the DRXs (except that for beam 3). DR2 only has one DRSU on it (LWA) and not enough contiguous space to record 4 hours of DRX. TBN gains looks a little high considering the images coming out of PASI. LS001, block 1. DRXs went through, DR4 didn't start recording due to invalid time. For the record I sent a SYN to all 5 DR machines before the start of the observations. Beam 2 not used due to the DRSU storage/missing VT DRSU. Hardware problems are suspected. Reset TBN gain to 20 at 5:15 UT. No other problems tonight. Very strange that this observation is still running. The mselog.txt indicates that the source has set. I should probably try to stop the recordings. TBN restarted at ~15:30 UT. Frequency of TBN changed at ~16:30 UT. All recording stopped at 16:33 UT. Jupiter: Had to restart executive a couple of times. Manually logged into DR4 and ran "sudo ntpdate -u -d 10.1.1.50" to get the time updated. It was ~6 seconds off. All observations started at 00:35 UT (5 minutes late). DR4 and 5 rejected due to insufficient drive space. DR4 adjusted down to 5,170 seconds to get it to work. MGT underflow on beam 3 by 2:45 UT. SDP: Scheduled beams 1, 3, and 4. TBN running on DR5 with only 7,480 seconds of recording time due to disk space. DR4 rejected REC again. Beams seem to have taken the DRX/BAM commands without problem. Restarted TBN with gain of 20 at 5:11 UT. LE002, Block 6: Greg was able to get the VT DRSU up on DR2 so we should be go for four beams tonight. Manually adjusted the time in DR4 to try to avoid the invalid time problem. DR1, 3, and 4 rejected due to insufficient space. Restarted at 00:34 UT using 6,140 seconds. DR5 rejected due to insufficient space. Restarted at 00:34 UT with 1,140 seconds. TBN gain looks to have been set to 20 this go around with the new code. Reset TBN at ~3:30 UT because the images looked pretty bad. Images were good afterwards until BAM hit. Lots of under/overflows in board.log file from TBN. Looking pretty bad at ~3:32 UT and restarted TBN again. DR spec. Test: Almost worked except I didn't change frequencies down to something low and DR1 rejected the REC with "operation not permitted". Try again later, say 3:45 UT. Retried at 3:50 UT with DR2 and 3. Seems to have worked alright. The DR2 file is copying over. Seems to be good agreement between DR spec. and LSL. Tried a TBN BAM test. Got PASI running at 74.03 MHz and everything looked good. Sent the BAM command: ./mesix DP_ BAM "1 740_706_0095.df 111226_XY.gf 0" and verified timetag problems in PASI and that the images changed. Probably shouldn't call it timetag problems, more like synchronization problems. Captured a 60 second TBN snippet with DR5 for a detailed look. Test 2: Do an INI and send the same BAM command after verifying that the PASI images look good again. Similar change as with the previous test in terms of image quality. Test 3: Do an INI and send a BAM command with an old .df file: ./mesix DP_ BAM "1 beams_74MHz_0az_83el_100bg.df 111226_XY.gf 0" Same results. Test 4: Do an INI and send a BAM with an old .gf file: ./mesix DP_ BAM "1 740_706_0095.df beams_74MHz_0az_83el_100bg.gf 0" Same results. Test 5: Do an INI and send an old BAM (.df and .gf): ./mesix DP_ BAM "1 beams_74MHz_0az_83el_100bg.df beams_74MHz_0az_83el_100bg.gf 0" Same results. Test 6: Switch back to the older version of the control software and try #5 again. Same results. Test 7: Same as #6 but using the command #2. Same results.