TBN mode majority of the day 130108 - Observation LH006 failed to write data, PEBKAC, wasted 3 hours 130108 18:38:31 56300 67111198 T 157168 3 DP_ RPT T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected| Current draw looked okay. Ran INIdp.sh, but did not run startTBN and all sky imaging was doen for a few hours. TBN mode majority of the day 130109 01:33:31 56301 5611273 T 158589 3 DP_ RPT T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected| INIdp.sh fixed. Status on webpage appeared to be stuck, showing BOOTING for DP_ well after it was up and TBN was working and update time on webpage should have shown NORMAL per RPT SUMMARY queries. A later INIdp.sh fixed status info. 130109 04:30 dp INI'd off, performed TBW capture to check antenna health 130109 04:45 set ASP to FULL in prep for Jupiter obs LH004 dos2unix'd Adjusted B4 SDF to run on beam 4, was set to DRX_BEAM 3, which would have collided with B3 SDF LH004 101 056301_000159320 Filesize: 1443562834272 LH004 102 056301_000159321 Filesize: 1443524101248 LH004 103 056301_000159322 Filesize: 1443557979744 130109 16:00 Woke up to 130109 15:58:31 56301 57511471 T 162674 3 DP_ RPT T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected| Appears to have started: 130109 10:13:31 56301 36811460 T 161493 3 DP_ RPT T_NOM1 T_NOM2 T_NOM3 T_NOM4! 0x06! Beamformer calibration failed| (right after the end of LH004) 130109 16:30 TBW antenna health check shows 72 bad dipoles, with many showing up good in current SSMIF Issuing ASP SHT/INI/FPW sequence Took two times to fix. Later in week was advised to slow down on issuing ASP related commands and this seemed to help. 130109 20:30 - 22:30 Joe working on site 130109 00:05 - 02:05 Jake working on PASI TBN mode majority of the day 130110 00:05 - 02:05 Jake H working on PASI 130110 07:45 high temp event on FPGAs, peaked at around 84.2C with outside temp -13 C. Normal? Frank advised slow start on control loop for temp control in room. LH006 41 Tag 056302_000166007 Filesize: 8839117416 LH006 42 Tag 056302_000166008 Filesize: 8838920580 LH006 43 Tag 056302_000166009 Filesize: 189159396 # low data LH006 44 Tag 056302_000166010 Filesize: 1326477804 # low data TBN mode majority of the day 130111 07:45 LH006 45 056303_000173793 size: 8839117416 LH006 46 056303_000173794 size: 8839314252 LH006 47 056303_000173792 size: 0 # low data Plenty of space LH006 48 056303_000173791 size: 1295574552 # low data 130111 16:23:32 56303 59012262 T 178456 3 DP_ RPT T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected| SHT/INI/FPW fixed 130111 18:05 LW001 1042 056303_000178829 35378702856 LW001 1043 056303_000178830 35378932764 LW001 1044 056303_000178831 19452975696 # low? LW001 1045 056303_000178832 35378702856 TBN mode majority of the day 130112 01:21 When performing a runLatestCheck.sh for acquireTBW.py (system looked good from TBN all sky image and histograms): ... Performing antenna health test for 130112_005911_tbwspecs.dat ... AntennaHealthTest.py:69: RuntimeWarning: divide by zero encountered in log10 spec = 10.*np.log10(spec) # convert to dB ... Many stands turned up bad, re-ran ASP INI 33 and then turned frontends back on. 130112 07:40 - ASP showed error, reset it. This may have caused problems for the current observation. 130112 07:45 LH006 49 056304_000182511 8839314252 LH006 50 056304_000182512 8839511088 LH006 51 056304_000182513 257461488 # low data LH006 52 056304_000182514 1308368892 # low data 130112 17:43:32 56304 63812556 T 187474 3 DP_ RPT T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected| ... 130112 22:38:32 56304 81512606 T 188509 3 DP_ RPT T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected| TBN mode majority of the day 130112 07:45 LH006 53 056304_000182511 8839314252 LH006 54 056304_000182512 8839511088 LH006 55 056304_000182513 257461488 # low data LH006 56 056304_000182514 1308368892 # low data 130113 Ran solar observations despite some bad stands Ran through ASP/DP_ SHT/INI/FPW etc.., different stands showed up bad after TBW collections. Advice later was to stretch out flow of ASP commands This seems to help, adding ~60s between ASP commands -> ./mesix ASP FPW 000111 && sleep 60 .. 130113 18:35 LW001 1046 056305_000195894 32432661744 LW001 1047 056305_000195895 32433351468 LW001 1048 056305_000195896 17793959568 LW001 1049 056305_000195897 32432891652 TBN mode majority of the day 130112 07:45 LH006 57 056306_000199347 8839117416 LH006 58 056306_000199348 8838920580 LH006 59 056306_000199349 266909616 # low data LH006 60 056306_000199350 1234161720 # low data 130113 18:35 LW001 1050 056306_000204464 32431972020 LW001 1051 056306_000204465 32432201928 LW001 1052 056306_000204466 17893049916 LW001 1053 056306_000204467 32431742112 130115 01:44 - 04:44 Jake working on PASI