2018-11-19 13:25:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:25:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:25:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:25:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:25:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:25:05 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:25:05 [DEBUG   ] Finished updating board flags in 1.941 seconds
2018-11-19 13:25:13 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 13:25:33 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:25:46 [DEBUG   ] Got command RPT from MCS: ref #559033
2018-11-19 13:25:46 [DEBUG   ] summary = ERROR
2018-11-19 13:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559033  13 58441 48346393 A  ERRORERROR'
2018-11-19 13:25:46 [DEBUG   ] Got command RPT from MCS: ref #559034
2018-11-19 13:25:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559034  77 58441 48346394 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:25:53 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:26:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:26:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:26:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:26:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:26:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:26:05 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:26:05 [DEBUG   ] Finished updating board flags in 1.856 seconds
2018-11-19 13:26:13 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:26:33 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:26:53 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:27:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:27:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:27:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:27:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:27:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:27:05 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:27:05 [DEBUG   ] Finished updating board flags in 1.760 seconds
2018-11-19 13:27:13 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:27:33 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:27:53 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:28:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:28:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 13:28:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:28:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:28:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:28:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:28:05 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:28:05 [DEBUG   ] Finished updating board flags in 1.625 seconds
2018-11-19 13:28:13 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:28:33 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:28:53 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:29:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:29:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 13:29:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:29:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:29:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:29:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:29:05 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:29:05 [DEBUG   ] Finished updating board flags in 1.633 seconds
2018-11-19 13:29:13 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 13:29:33 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:29:52 [DEBUG   ] Got command RPT from MCS: ref #559056
2018-11-19 13:29:52 [DEBUG   ] summary = ERROR
2018-11-19 13:29:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559056  13 58441 48592505 A  ERRORERROR'
2018-11-19 13:29:52 [DEBUG   ] Got command RPT from MCS: ref #559057
2018-11-19 13:29:52 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:29:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559057  77 58441 48592709 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:29:52 [DEBUG   ] Got command RPT from MCS: ref #559058
2018-11-19 13:29:52 [DEBUG   ] summary = ERROR
2018-11-19 13:29:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559058  13 58441 48592902 A  ERRORERROR'
2018-11-19 13:29:53 [DEBUG   ] Got command RPT from MCS: ref #559060
2018-11-19 13:29:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:29:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559060  77 58441 48593105 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:29:53 [DEBUG   ] Got command RPT from MCS: ref #559062
2018-11-19 13:29:53 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 13:29:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559062  12 58441 48593309 A  ERRORB˜™š'
2018-11-19 13:29:53 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 13:30:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:30:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:30:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:30:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:30:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:30:05 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:30:05 [DEBUG   ] Finished updating board flags in 1.540 seconds
2018-11-19 13:30:13 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:30:33 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 13:30:46 [DEBUG   ] Got command RPT from MCS: ref #559066
2018-11-19 13:30:46 [DEBUG   ] summary = ERROR
2018-11-19 13:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559066  13 58441 48646393 A  ERRORERROR'
2018-11-19 13:30:46 [DEBUG   ] Got command RPT from MCS: ref #559067
2018-11-19 13:30:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559067  77 58441 48646394 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:30:53 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 13:31:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:31:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:31:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:31:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:31:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:31:05 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:31:05 [DEBUG   ] Finished updating board flags in 1.502 seconds
2018-11-19 13:31:14 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-19 13:31:34 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 13:31:54 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-19 13:32:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 13:32:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:32:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:32:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:32:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:32:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:32:05 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:32:05 [DEBUG   ] Finished updating board flags in 1.504 seconds
2018-11-19 13:32:14 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 13:32:34 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 13:32:54 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 13:33:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:33:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 13:33:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:33:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:33:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:33:05 [WARNING ] (13, 1, 'bot')
2018-11-19 13:33:05 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:33:05 [DEBUG   ] Finished updating board flags in 1.512 seconds
2018-11-19 13:33:14 [DEBUG   ] Finished updating ant stats in 0.997 seconds
2018-11-19 13:33:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 13:33:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 13:34:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:34:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:34:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:34:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:34:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:34:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:34:06 [DEBUG   ] Finished updating board flags in 2.123 seconds
2018-11-19 13:34:14 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 13:34:34 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 13:34:52 [DEBUG   ] Got command RPT from MCS: ref #559091
2018-11-19 13:34:52 [DEBUG   ] summary = ERROR
2018-11-19 13:34:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559091  13 58441 48892585 A  ERRORERROR'
2018-11-19 13:34:52 [DEBUG   ] Got command RPT from MCS: ref #559092
2018-11-19 13:34:52 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:34:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559092  77 58441 48892789 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:34:52 [DEBUG   ] Got command RPT from MCS: ref #559093
2018-11-19 13:34:52 [DEBUG   ] summary = ERROR
2018-11-19 13:34:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559093  13 58441 48892993 A  ERRORERROR'
2018-11-19 13:34:53 [DEBUG   ] Got command RPT from MCS: ref #559094
2018-11-19 13:34:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:34:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559094  77 58441 48893185 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:34:53 [DEBUG   ] Got command RPT from MCS: ref #559095
2018-11-19 13:34:53 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 13:34:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559095  12 58441 48893389 A  ERRORB˜™š'
2018-11-19 13:34:54 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:35:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:35:06 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 13:35:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:35:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:35:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:35:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:35:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:35:06 [DEBUG   ] Finished updating board flags in 1.971 seconds
2018-11-19 13:35:14 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:35:34 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:35:46 [DEBUG   ] Got command RPT from MCS: ref #559096
2018-11-19 13:35:46 [DEBUG   ] summary = ERROR
2018-11-19 13:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559096  13 58441 48946401 A  ERRORERROR'
2018-11-19 13:35:46 [DEBUG   ] Got command RPT from MCS: ref #559097
2018-11-19 13:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559097  77 58441 48946402 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:35:54 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:36:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:36:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:36:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:36:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:36:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:36:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:36:06 [DEBUG   ] Finished updating board flags in 1.854 seconds
2018-11-19 13:36:14 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:36:34 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:36:54 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 13:37:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:37:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 13:37:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:37:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:37:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:37:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:37:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:37:06 [DEBUG   ] Finished updating board flags in 1.814 seconds
2018-11-19 13:37:14 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:37:34 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:37:54 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:38:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:38:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:38:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:38:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:38:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:38:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:38:06 [DEBUG   ] Finished updating board flags in 1.747 seconds
2018-11-19 13:38:14 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 13:38:34 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:38:54 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:39:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:39:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:39:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:39:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:39:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:39:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:39:06 [DEBUG   ] Finished updating board flags in 1.654 seconds
2018-11-19 13:39:14 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 13:39:34 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-19 13:39:52 [DEBUG   ] Got command RPT from MCS: ref #559119
2018-11-19 13:39:52 [DEBUG   ] summary = ERROR
2018-11-19 13:39:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559119  13 58441 49192677 A  ERRORERROR'
2018-11-19 13:39:52 [DEBUG   ] Got command RPT from MCS: ref #559120
2018-11-19 13:39:52 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:39:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559120  77 58441 49192869 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:39:53 [DEBUG   ] Got command RPT from MCS: ref #559121
2018-11-19 13:39:53 [DEBUG   ] summary = ERROR
2018-11-19 13:39:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559121  13 58441 49193073 A  ERRORERROR'
2018-11-19 13:39:53 [DEBUG   ] Got command RPT from MCS: ref #559123
2018-11-19 13:39:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:39:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559123  77 58441 49193277 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:39:53 [DEBUG   ] Got command RPT from MCS: ref #559125
2018-11-19 13:39:53 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 13:39:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559125  12 58441 49193481 A  ERRORB˜™š'
2018-11-19 13:39:55 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-19 13:40:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:40:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:40:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:40:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:40:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:40:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:40:06 [DEBUG   ] Finished updating board flags in 1.541 seconds
2018-11-19 13:40:15 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 13:40:35 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 13:40:46 [DEBUG   ] Got command RPT from MCS: ref #559133
2018-11-19 13:40:46 [DEBUG   ] summary = ERROR
2018-11-19 13:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559133  13 58441 49246401 A  ERRORERROR'
2018-11-19 13:40:46 [DEBUG   ] Got command RPT from MCS: ref #559134
2018-11-19 13:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559134  77 58441 49246402 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:40:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 13:41:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:41:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:41:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:41:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:41:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:41:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:41:06 [DEBUG   ] Finished updating board flags in 1.471 seconds
2018-11-19 13:41:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 13:41:35 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 13:41:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 13:42:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:42:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:42:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:42:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:42:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:42:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:42:06 [DEBUG   ] Finished updating board flags in 1.496 seconds
2018-11-19 13:42:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 13:42:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 13:42:55 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 13:43:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 13:43:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:43:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:43:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:43:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:43:06 [WARNING ] (13, 1, 'bot')
2018-11-19 13:43:06 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:43:06 [DEBUG   ] Finished updating board flags in 1.522 seconds
2018-11-19 13:43:15 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 13:43:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 13:43:55 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 13:44:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:44:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:44:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:44:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:44:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:44:07 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:44:07 [DEBUG   ] Finished updating board flags in 2.185 seconds
2018-11-19 13:44:15 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 13:44:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 13:44:52 [DEBUG   ] Got command RPT from MCS: ref #559154
2018-11-19 13:44:52 [DEBUG   ] summary = ERROR
2018-11-19 13:44:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559154  13 58441 49492765 A  ERRORERROR'
2018-11-19 13:44:52 [DEBUG   ] Got command RPT from MCS: ref #559155
2018-11-19 13:44:52 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:44:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559155  77 58441 49492957 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:44:53 [DEBUG   ] Got command RPT from MCS: ref #559156
2018-11-19 13:44:53 [DEBUG   ] summary = ERROR
2018-11-19 13:44:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559156  13 58441 49493161 A  ERRORERROR'
2018-11-19 13:44:53 [DEBUG   ] Got command RPT from MCS: ref #559157
2018-11-19 13:44:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:44:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559157  77 58441 49493365 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:44:53 [DEBUG   ] Got command RPT from MCS: ref #559158
2018-11-19 13:44:53 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 13:44:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559158  12 58441 49493569 A  ERRORB™™š'
2018-11-19 13:44:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 13:45:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:45:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:45:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:45:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:45:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:45:07 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:45:07 [DEBUG   ] Finished updating board flags in 1.952 seconds
2018-11-19 13:45:15 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 13:45:35 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 13:45:46 [DEBUG   ] Got command RPT from MCS: ref #559159
2018-11-19 13:45:46 [DEBUG   ] summary = ERROR
2018-11-19 13:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559159  13 58441 49546397 A  ERRORERROR'
2018-11-19 13:45:46 [DEBUG   ] Got command RPT from MCS: ref #559160
2018-11-19 13:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559160  77 58441 49546398 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:45:55 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 13:46:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:46:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:46:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:46:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:46:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:46:07 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:46:07 [DEBUG   ] Finished updating board flags in 1.829 seconds
2018-11-19 13:46:15 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 13:46:35 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 13:46:55 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 13:47:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:47:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:47:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:47:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:47:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:47:07 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:47:07 [DEBUG   ] Finished updating board flags in 1.752 seconds
2018-11-19 13:47:15 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 13:47:35 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 13:47:55 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 13:48:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 13:48:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:48:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:48:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:48:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:48:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:48:07 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:48:07 [DEBUG   ] Finished updating board flags in 1.620 seconds
2018-11-19 13:48:15 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 13:48:35 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 13:48:55 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-19 13:49:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:49:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:49:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:49:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:49:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:49:07 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:49:07 [DEBUG   ] Finished updating board flags in 1.597 seconds
2018-11-19 13:49:15 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 13:49:36 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-19 13:49:52 [DEBUG   ] Got command RPT from MCS: ref #559186
2018-11-19 13:49:52 [DEBUG   ] summary = ERROR
2018-11-19 13:49:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559186  13 58441 49792841 A  ERRORERROR'
2018-11-19 13:49:53 [DEBUG   ] Got command RPT from MCS: ref #559187
2018-11-19 13:49:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:49:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559187  77 58441 49793045 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:49:53 [DEBUG   ] Got command RPT from MCS: ref #559188
2018-11-19 13:49:53 [DEBUG   ] summary = ERROR
2018-11-19 13:49:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559188  13 58441 49793249 A  ERRORERROR'
2018-11-19 13:49:53 [DEBUG   ] Got command RPT from MCS: ref #559190
2018-11-19 13:49:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:49:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559190  77 58441 49793441 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:49:53 [DEBUG   ] Got command RPT from MCS: ref #559191
2018-11-19 13:49:53 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 13:49:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559191  12 58441 49793645 A  ERRORB˜™š'
2018-11-19 13:49:56 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 13:50:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:50:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:50:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:50:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:50:07 [WARNING ] (13, 1, 'bot')
2018-11-19 13:50:07 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:50:07 [DEBUG   ] Finished updating board flags in 1.396 seconds
2018-11-19 13:50:16 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 13:50:36 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 13:50:46 [DEBUG   ] Got command RPT from MCS: ref #559196
2018-11-19 13:50:46 [DEBUG   ] summary = ERROR
2018-11-19 13:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559196  13 58441 49846405 A  ERRORERROR'
2018-11-19 13:50:46 [DEBUG   ] Got command RPT from MCS: ref #559197
2018-11-19 13:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559197  77 58441 49846406 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:50:56 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 13:51:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:51:08 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 13:51:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:51:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:51:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:51:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:51:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:51:08 [DEBUG   ] Finished updating board flags in 2.207 seconds
2018-11-19 13:51:16 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 13:51:36 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 13:51:56 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 13:52:00 [DEBUG   ] Got command TBN from MCS: ref #559217
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa27', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '1', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa24', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '2', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] tbn = exited with status True, code 0x00
2018-11-19 13:52:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_TBN   559217   8 58441 49920522 A  ERROR'
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa23', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '3', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa18', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '4', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa10', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '5', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa32', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '6', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa11', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '7', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa30', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '8', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa12', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '9', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa8', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '10', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa13', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '11', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa26', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '12', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa14', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '13', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa22', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '14', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa20', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '15', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa1', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '16', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa5', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '17', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa25', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '18', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa35', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '19', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa19', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '20', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa21', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '21', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa6', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '22', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa7', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '23', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa33', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '24', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa15', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '25', '1542635522']
2018-11-19 13:52:00 [DEBUG   ] spawn_process: at 2018-11-19 13:52:00 running ['/usr/local/bin/lwa28', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '26', '1542635522']
2018-11-19 13:52:00 [INFO    ] Finished spawning TBN in 0.052 s
2018-11-19 13:52:07 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 13:52:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:52:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:52:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:52:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:52:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:52:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:52:08 [DEBUG   ] Finished updating board flags in 2.184 seconds
2018-11-19 13:52:16 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 13:52:36 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 13:52:56 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 13:53:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:53:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:53:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:53:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:53:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:53:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:53:08 [DEBUG   ] Finished updating board flags in 2.113 seconds
2018-11-19 13:53:16 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 13:53:36 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 13:53:56 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 13:54:07 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 13:54:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:54:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 13:54:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:54:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:54:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:54:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:54:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:54:08 [DEBUG   ] Finished updating board flags in 1.903 seconds
2018-11-19 13:54:16 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 13:54:36 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 13:54:52 [DEBUG   ] Got command RPT from MCS: ref #559218
2018-11-19 13:54:52 [DEBUG   ] summary = ERROR
2018-11-19 13:54:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559218  13 58441 50092933 A  ERRORERROR'
2018-11-19 13:54:53 [DEBUG   ] Got command RPT from MCS: ref #559219
2018-11-19 13:54:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:54:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559219  77 58441 50093137 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:54:53 [DEBUG   ] Got command RPT from MCS: ref #559220
2018-11-19 13:54:53 [DEBUG   ] summary = ERROR
2018-11-19 13:54:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559220  13 58441 50093329 A  ERRORERROR'
2018-11-19 13:54:53 [DEBUG   ] Got command RPT from MCS: ref #559221
2018-11-19 13:54:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:54:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559221  77 58441 50093533 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:54:53 [DEBUG   ] Got command RPT from MCS: ref #559222
2018-11-19 13:54:53 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 13:54:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559222  12 58441 50093737 A  ERRORB™™š'
2018-11-19 13:54:56 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 13:55:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:55:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:55:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:55:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:55:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:55:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:55:08 [DEBUG   ] Finished updating board flags in 1.823 seconds
2018-11-19 13:55:16 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 13:55:36 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 13:55:46 [DEBUG   ] Got command RPT from MCS: ref #559227
2018-11-19 13:55:46 [DEBUG   ] summary = ERROR
2018-11-19 13:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559227  13 58441 50146401 A  ERRORERROR'
2018-11-19 13:55:46 [DEBUG   ] Got command RPT from MCS: ref #559228
2018-11-19 13:55:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559228  77 58441 50146402 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:55:56 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 13:56:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:56:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:56:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:56:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:56:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:56:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:56:08 [DEBUG   ] Finished updating board flags in 1.786 seconds
2018-11-19 13:56:11 [DEBUG   ] Got command TBW from MCS: ref #559245
2018-11-19 13:56:11 [DEBUG   ] tbw = exited with status True, code 0x00
2018-11-19 13:56:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_TBW   559245   8 58441 50171590 A  ERROR'
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa27 command /usr/local/bin/lwa27 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa24 command /usr/local/bin/lwa24 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa23 command /usr/local/bin/lwa23 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa18 command /usr/local/bin/lwa18 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa10 command /usr/local/bin/lwa10 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa32 command /usr/local/bin/lwa32 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa11 command /usr/local/bin/lwa11 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa30 command /usr/local/bin/lwa30 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa12 command /usr/local/bin/lwa12 pkill tbn
2018-11-19 13:56:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa8 command /usr/local/bin/lwa8 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa13 command /usr/local/bin/lwa13 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa26 command /usr/local/bin/lwa26 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa14 command /usr/local/bin/lwa14 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa22 command /usr/local/bin/lwa22 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa20 command /usr/local/bin/lwa20 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa1 command /usr/local/bin/lwa1 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa5 command /usr/local/bin/lwa5 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa25 command /usr/local/bin/lwa25 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa35 command /usr/local/bin/lwa35 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa19 command /usr/local/bin/lwa19 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa21 command /usr/local/bin/lwa21 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa6 command /usr/local/bin/lwa6 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa7 command /usr/local/bin/lwa7 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa33 command /usr/local/bin/lwa33 pkill tbn
2018-11-19 13:56:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa15 command /usr/local/bin/lwa15 pkill tbn
2018-11-19 13:56:15 [DEBUG   ] sendHostnameCommand - Sent hostname lwa28 command /usr/local/bin/lwa28 pkill tbn
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa27', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '1', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa24', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '2', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa23', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '3', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa18', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '4', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa10', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '5', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa32', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '6', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa11', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '7', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa30', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '8', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa12', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '9', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa8', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '10', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa13', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '11', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa26', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '12', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa14', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '13', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa22', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '14', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa20', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '15', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa1', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '16', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa5', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '17', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa25', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '18', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa35', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '19', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa19', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '20', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa21', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '21', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa6', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '22', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa7', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '23', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa33', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '24', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa15', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '25', '12000000']
2018-11-19 13:56:15 [DEBUG   ] spawn_process: at 2018-11-19 13:56:15 running ['/usr/local/bin/lwa28', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '26', '12000000']
2018-11-19 13:56:15 [INFO    ] Finished spawning TBW
2018-11-19 13:57:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:57:08 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 13:57:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:57:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:57:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:57:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:57:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:57:08 [DEBUG   ] Finished updating board flags in 1.560 seconds
2018-11-19 13:58:03 [INFO    ] __waitForTBW - TBW finished in 108.638 s
2018-11-19 13:58:04 [DEBUG   ] Finished updating ant stats in 0.534 seconds
2018-11-19 13:58:04 [INFO    ] Started antennas statistics background thread
2018-11-19 13:58:04 [INFO    ] Resuming TBN at 38.100 MHz after TBW acquisition
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa27', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '1', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa24', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '2', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa23', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '3', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa18', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '4', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa10', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '5', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa32', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '6', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa11', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '7', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa30', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '8', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa12', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '9', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa8', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '10', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa13', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '11', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa26', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '12', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa14', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '13', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa22', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '14', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa20', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '15', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa1', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '16', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa5', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '17', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa25', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '18', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa35', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '19', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa19', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '20', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa21', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '21', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa6', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '22', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa7', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '23', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa33', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '24', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa15', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '25', '1542635886']
2018-11-19 13:58:04 [DEBUG   ] spawn_process: at 2018-11-19 13:58:04 running ['/usr/local/bin/lwa28', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '26', '1542635886']
2018-11-19 13:58:04 [INFO    ] Finished spawning TBN in 0.050 s
2018-11-19 13:58:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 13:58:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:58:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 13:58:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:58:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:58:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:58:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:58:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:58:08 [DEBUG   ] Finished updating board flags in 1.248 seconds
2018-11-19 13:58:24 [DEBUG   ] Finished updating ant stats in 0.467 seconds
2018-11-19 13:58:44 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-19 13:59:05 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-19 13:59:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 13:59:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 13:59:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:59:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:59:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:59:08 [WARNING ] (13, 1, 'bot')
2018-11-19 13:59:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 13:59:08 [DEBUG   ] Finished updating board flags in 1.492 seconds
2018-11-19 13:59:25 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 13:59:45 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 13:59:53 [DEBUG   ] Got command RPT from MCS: ref #559259
2018-11-19 13:59:53 [DEBUG   ] summary = ERROR
2018-11-19 13:59:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559259  13 58441 50393013 A  ERRORERROR'
2018-11-19 13:59:53 [DEBUG   ] Got command RPT from MCS: ref #559260
2018-11-19 13:59:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:59:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559260  77 58441 50393217 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:59:53 [DEBUG   ] Got command RPT from MCS: ref #559261
2018-11-19 13:59:53 [DEBUG   ] summary = ERROR
2018-11-19 13:59:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559261  13 58441 50393421 A  ERRORERROR'
2018-11-19 13:59:53 [DEBUG   ] Got command RPT from MCS: ref #559263
2018-11-19 13:59:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 13:59:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559263  77 58441 50393625 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 13:59:53 [DEBUG   ] Got command RPT from MCS: ref #559264
2018-11-19 13:59:53 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 13:59:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559264  12 58441 50393817 A  ERRORB˜™š'
2018-11-19 14:00:05 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 14:00:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:00:08 [WARNING ] (13, 1, 'bot')
2018-11-19 14:00:08 [WARNING ] (13, 1, 'bot')
2018-11-19 14:00:08 [WARNING ] (13, 1, 'bot')
2018-11-19 14:00:08 [WARNING ] (13, 1, 'bot')
2018-11-19 14:00:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:00:08 [DEBUG   ] Finished updating board flags in 1.484 seconds
2018-11-19 14:00:25 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 14:00:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:00:46 [DEBUG   ] Got command RPT from MCS: ref #559269
2018-11-19 14:00:46 [DEBUG   ] summary = ERROR
2018-11-19 14:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559269  13 58441 50446405 A  ERRORERROR'
2018-11-19 14:00:46 [DEBUG   ] Got command RPT from MCS: ref #559270
2018-11-19 14:00:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559270  77 58441 50446406 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:01:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:01:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:01:08 [WARNING ] (13, 1, 'bot')
2018-11-19 14:01:08 [WARNING ] (13, 1, 'bot')
2018-11-19 14:01:08 [WARNING ] (13, 1, 'bot')
2018-11-19 14:01:08 [WARNING ] (13, 1, 'bot')
2018-11-19 14:01:08 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:01:08 [DEBUG   ] Finished updating board flags in 1.487 seconds
2018-11-19 14:01:25 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 14:01:45 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 14:02:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 14:02:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:02:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:02:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:02:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:02:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:02:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:02:09 [DEBUG   ] Finished updating board flags in 2.157 seconds
2018-11-19 14:02:25 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 14:02:45 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 14:03:05 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 14:03:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:03:09 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 14:03:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:03:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:03:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:03:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:03:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:03:09 [DEBUG   ] Finished updating board flags in 1.972 seconds
2018-11-19 14:03:25 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 14:03:45 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 14:04:05 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 14:04:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:04:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:04:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:04:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:04:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:04:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:04:09 [DEBUG   ] Finished updating board flags in 1.827 seconds
2018-11-19 14:04:25 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 14:04:45 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:04:53 [DEBUG   ] Got command RPT from MCS: ref #559294
2018-11-19 14:04:53 [DEBUG   ] summary = ERROR
2018-11-19 14:04:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559294  13 58441 50693101 A  ERRORERROR'
2018-11-19 14:04:53 [DEBUG   ] Got command RPT from MCS: ref #559295
2018-11-19 14:04:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:04:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559295  77 58441 50693305 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:04:53 [DEBUG   ] Got command RPT from MCS: ref #559296
2018-11-19 14:04:53 [DEBUG   ] summary = ERROR
2018-11-19 14:04:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559296  13 58441 50693509 A  ERRORERROR'
2018-11-19 14:04:53 [DEBUG   ] Got command RPT from MCS: ref #559297
2018-11-19 14:04:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:04:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559297  77 58441 50693701 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:04:53 [DEBUG   ] Got command RPT from MCS: ref #559298
2018-11-19 14:04:53 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:04:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559298  12 58441 50693905 A  ERRORB™™š'
2018-11-19 14:05:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 14:05:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:05:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 14:05:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:05:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:05:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:05:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:05:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:05:09 [DEBUG   ] Finished updating board flags in 1.787 seconds
2018-11-19 14:05:25 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 14:05:45 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 14:05:46 [DEBUG   ] Got command RPT from MCS: ref #559299
2018-11-19 14:05:46 [DEBUG   ] summary = ERROR
2018-11-19 14:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559299  13 58441 50746401 A  ERRORERROR'
2018-11-19 14:05:46 [DEBUG   ] Got command RPT from MCS: ref #559300
2018-11-19 14:05:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559300  77 58441 50746402 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:06:05 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:06:08 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 14:06:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:06:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:06:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:06:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:06:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:06:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:06:09 [DEBUG   ] Finished updating board flags in 1.616 seconds
2018-11-19 14:06:25 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:06:45 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 14:07:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 14:07:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:07:09 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 14:07:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:07:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:07:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:07:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:07:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:07:09 [DEBUG   ] Finished updating board flags in 1.702 seconds
2018-11-19 14:07:25 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 14:07:46 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-19 14:08:06 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 14:08:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:08:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:08:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:08:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:08:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:08:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:08:09 [DEBUG   ] Finished updating board flags in 1.483 seconds
2018-11-19 14:08:26 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:08:46 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:09:06 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:09:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:09:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:09:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:09:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:09:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:09:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:09:09 [DEBUG   ] Finished updating board flags in 1.423 seconds
2018-11-19 14:09:26 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:09:46 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:09:53 [DEBUG   ] Got command RPT from MCS: ref #559322
2018-11-19 14:09:53 [DEBUG   ] summary = ERROR
2018-11-19 14:09:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559322  13 58441 50993189 A  ERRORERROR'
2018-11-19 14:09:53 [DEBUG   ] Got command RPT from MCS: ref #559323
2018-11-19 14:09:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:09:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559323  77 58441 50993393 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:09:53 [DEBUG   ] Got command RPT from MCS: ref #559324
2018-11-19 14:09:53 [DEBUG   ] summary = ERROR
2018-11-19 14:09:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559324  13 58441 50993585 A  ERRORERROR'
2018-11-19 14:09:53 [DEBUG   ] Got command RPT from MCS: ref #559326
2018-11-19 14:09:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:09:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559326  77 58441 50993789 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:09:53 [DEBUG   ] Got command RPT from MCS: ref #559327
2018-11-19 14:09:53 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:09:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559327  12 58441 50993993 A  ERRORB˜™š'
2018-11-19 14:10:06 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:10:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:10:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:10:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:10:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:10:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:10:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:10:09 [DEBUG   ] Finished updating board flags in 1.396 seconds
2018-11-19 14:10:26 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:10:46 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 14:10:46 [DEBUG   ] Got command RPT from MCS: ref #559336
2018-11-19 14:10:46 [DEBUG   ] summary = ERROR
2018-11-19 14:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559336  13 58441 51046405 A  ERRORERROR'
2018-11-19 14:10:46 [DEBUG   ] Got command RPT from MCS: ref #559337
2018-11-19 14:10:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559337  77 58441 51046406 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:11:06 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:11:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:11:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:11:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:11:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:11:09 [WARNING ] (13, 1, 'bot')
2018-11-19 14:11:09 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:11:09 [DEBUG   ] Finished updating board flags in 1.456 seconds
2018-11-19 14:11:26 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:11:46 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:12:06 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 14:12:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:12:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:12:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:12:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:12:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:12:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:12:10 [DEBUG   ] Finished updating board flags in 2.246 seconds
2018-11-19 14:12:26 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:12:46 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:13:06 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:13:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:13:10 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 14:13:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:13:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:13:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:13:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:13:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:13:10 [DEBUG   ] Finished updating board flags in 2.215 seconds
2018-11-19 14:13:26 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:13:46 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:14:06 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:14:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:14:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:14:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:14:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:14:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:14:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:14:10 [DEBUG   ] Finished updating board flags in 2.090 seconds
2018-11-19 14:14:26 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:14:46 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 14:14:53 [DEBUG   ] Got command RPT from MCS: ref #559367
2018-11-19 14:14:53 [DEBUG   ] summary = ERROR
2018-11-19 14:14:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559367  13 58441 51293277 A  ERRORERROR'
2018-11-19 14:14:53 [DEBUG   ] Got command RPT from MCS: ref #559368
2018-11-19 14:14:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:14:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559368  77 58441 51293469 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:14:53 [DEBUG   ] Got command RPT from MCS: ref #559369
2018-11-19 14:14:53 [DEBUG   ] summary = ERROR
2018-11-19 14:14:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559369  13 58441 51293673 A  ERRORERROR'
2018-11-19 14:14:53 [DEBUG   ] Got command RPT from MCS: ref #559370
2018-11-19 14:14:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:14:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559370  77 58441 51293877 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:14:54 [DEBUG   ] Got command RPT from MCS: ref #559371
2018-11-19 14:14:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:14:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559371  12 58441 51294081 A  ERRORB˜™š'
2018-11-19 14:15:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:15:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:15:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:15:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:15:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:15:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:15:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:15:10 [DEBUG   ] Finished updating board flags in 2.034 seconds
2018-11-19 14:15:26 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:15:46 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 14:15:46 [DEBUG   ] Got command RPT from MCS: ref #559372
2018-11-19 14:15:46 [DEBUG   ] summary = ERROR
2018-11-19 14:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559372  13 58441 51346405 A  ERRORERROR'
2018-11-19 14:15:46 [DEBUG   ] Got command RPT from MCS: ref #559373
2018-11-19 14:15:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559373  77 58441 51346406 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:16:06 [DEBUG   ] Finished updating ant stats in 0.997 seconds
2018-11-19 14:16:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:16:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:16:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:16:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:16:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:16:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:16:10 [DEBUG   ] Finished updating board flags in 1.958 seconds
2018-11-19 14:16:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 14:16:46 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 14:17:06 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-19 14:17:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:17:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:17:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:17:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:17:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:17:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:17:10 [DEBUG   ] Finished updating board flags in 1.824 seconds
2018-11-19 14:17:26 [DEBUG   ] Finished updating ant stats in 0.597 seconds
2018-11-19 14:17:46 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 14:18:06 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 14:18:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:18:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:18:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:18:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:18:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:18:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:18:10 [DEBUG   ] Finished updating board flags in 1.784 seconds
2018-11-19 14:18:27 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-19 14:18:47 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 14:19:07 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:19:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:19:10 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 14:19:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:19:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:19:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:19:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:19:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:19:10 [DEBUG   ] Finished updating board flags in 1.616 seconds
2018-11-19 14:19:27 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 14:19:47 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:19:53 [DEBUG   ] Got command RPT from MCS: ref #559399
2018-11-19 14:19:53 [DEBUG   ] summary = ERROR
2018-11-19 14:19:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559399  13 58441 51593357 A  ERRORERROR'
2018-11-19 14:19:53 [DEBUG   ] Got command RPT from MCS: ref #559400
2018-11-19 14:19:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:19:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559400  77 58441 51593561 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:19:53 [DEBUG   ] Got command RPT from MCS: ref #559401
2018-11-19 14:19:53 [DEBUG   ] summary = ERROR
2018-11-19 14:19:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559401  13 58441 51593765 A  ERRORERROR'
2018-11-19 14:19:53 [DEBUG   ] Got command RPT from MCS: ref #559403
2018-11-19 14:19:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:19:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559403  77 58441 51593957 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:19:54 [DEBUG   ] Got command RPT from MCS: ref #559405
2018-11-19 14:19:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:19:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559405  12 58441 51594161 A  ERRORB˜™š'
2018-11-19 14:20:07 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:20:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:20:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:20:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:20:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:20:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:20:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:20:10 [DEBUG   ] Finished updating board flags in 1.591 seconds
2018-11-19 14:20:27 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 14:20:46 [DEBUG   ] Got command RPT from MCS: ref #559409
2018-11-19 14:20:46 [DEBUG   ] summary = ERROR
2018-11-19 14:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559409  13 58441 51646413 A  ERRORERROR'
2018-11-19 14:20:46 [DEBUG   ] Got command RPT from MCS: ref #559410
2018-11-19 14:20:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559410  77 58441 51646414 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:20:47 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 14:21:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 14:21:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:21:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:21:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:21:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:21:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:21:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:21:10 [DEBUG   ] Finished updating board flags in 1.420 seconds
2018-11-19 14:21:27 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 14:21:47 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 14:22:07 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:22:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:22:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:22:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:22:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:22:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:22:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:22:10 [DEBUG   ] Finished updating board flags in 1.392 seconds
2018-11-19 14:22:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:22:47 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:23:07 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 14:23:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:23:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:23:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:23:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:23:10 [WARNING ] (13, 1, 'bot')
2018-11-19 14:23:10 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:23:10 [DEBUG   ] Finished updating board flags in 1.387 seconds
2018-11-19 14:23:27 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 14:23:48 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-19 14:24:08 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 14:24:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:24:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:24:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:24:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:24:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:24:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:24:11 [DEBUG   ] Finished updating board flags in 2.035 seconds
2018-11-19 14:24:28 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:24:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:24:53 [DEBUG   ] Got command RPT from MCS: ref #559430
2018-11-19 14:24:53 [DEBUG   ] summary = ERROR
2018-11-19 14:24:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559430  13 58441 51893445 A  ERRORERROR'
2018-11-19 14:24:53 [DEBUG   ] Got command RPT from MCS: ref #559431
2018-11-19 14:24:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:24:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559431  77 58441 51893649 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:24:53 [DEBUG   ] Got command RPT from MCS: ref #559432
2018-11-19 14:24:53 [DEBUG   ] summary = ERROR
2018-11-19 14:24:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559432  13 58441 51893853 A  ERRORERROR'
2018-11-19 14:24:54 [DEBUG   ] Got command RPT from MCS: ref #559433
2018-11-19 14:24:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:24:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559433  77 58441 51894045 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:24:54 [DEBUG   ] Got command RPT from MCS: ref #559434
2018-11-19 14:24:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:24:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559434  12 58441 51894249 A  ERRORB˜™š'
2018-11-19 14:25:08 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:25:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:25:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:25:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:25:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:25:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:25:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:25:11 [DEBUG   ] Finished updating board flags in 2.012 seconds
2018-11-19 14:25:28 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:25:46 [DEBUG   ] Got command RPT from MCS: ref #559439
2018-11-19 14:25:46 [DEBUG   ] summary = ERROR
2018-11-19 14:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559439  13 58441 51946405 A  ERRORERROR'
2018-11-19 14:25:46 [DEBUG   ] Got command RPT from MCS: ref #559440
2018-11-19 14:25:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559440  77 58441 51946406 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:25:48 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:26:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:26:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:26:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:26:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:26:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:26:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:26:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:26:11 [DEBUG   ] Finished updating board flags in 1.976 seconds
2018-11-19 14:26:28 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:26:48 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:27:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:27:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:27:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:27:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:27:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:27:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:27:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:27:11 [DEBUG   ] Finished updating board flags in 1.803 seconds
2018-11-19 14:27:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:27:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:28:08 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:28:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:28:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:28:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:28:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:28:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:28:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:28:11 [DEBUG   ] Finished updating board flags in 1.925 seconds
2018-11-19 14:28:28 [DEBUG   ] Finished updating ant stats in 1.191 seconds
2018-11-19 14:28:48 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 14:29:08 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 14:29:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:29:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:29:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:29:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:29:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:29:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:29:11 [DEBUG   ] Finished updating board flags in 1.712 seconds
2018-11-19 14:29:28 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 14:29:48 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:29:53 [DEBUG   ] Got command RPT from MCS: ref #559462
2018-11-19 14:29:53 [DEBUG   ] summary = ERROR
2018-11-19 14:29:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559462  13 58441 52193529 A  ERRORERROR'
2018-11-19 14:29:53 [DEBUG   ] Got command RPT from MCS: ref #559463
2018-11-19 14:29:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:29:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559463  77 58441 52193733 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:29:53 [DEBUG   ] Got command RPT from MCS: ref #559464
2018-11-19 14:29:53 [DEBUG   ] summary = ERROR
2018-11-19 14:29:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559464  13 58441 52193937 A  ERRORERROR'
2018-11-19 14:29:54 [DEBUG   ] Got command RPT from MCS: ref #559466
2018-11-19 14:29:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:29:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559466  77 58441 52194129 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:29:54 [DEBUG   ] Got command RPT from MCS: ref #559467
2018-11-19 14:29:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:29:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559467  12 58441 52194333 A  ERRORB˜™š'
2018-11-19 14:30:08 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:30:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:30:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:30:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:30:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:30:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:30:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:30:11 [DEBUG   ] Finished updating board flags in 1.569 seconds
2018-11-19 14:30:28 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 14:30:46 [DEBUG   ] Got command RPT from MCS: ref #559472
2018-11-19 14:30:46 [DEBUG   ] summary = ERROR
2018-11-19 14:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559472  13 58441 52246405 A  ERRORERROR'
2018-11-19 14:30:46 [DEBUG   ] Got command RPT from MCS: ref #559473
2018-11-19 14:30:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559473  77 58441 52246406 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:30:48 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:31:08 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:31:10 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 14:31:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:31:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:31:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:31:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:31:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:31:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:31:11 [DEBUG   ] Finished updating board flags in 1.473 seconds
2018-11-19 14:31:28 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 14:31:48 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:32:08 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:32:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 14:32:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:32:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:32:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:32:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:32:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:32:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:32:11 [DEBUG   ] Finished updating board flags in 1.470 seconds
2018-11-19 14:32:28 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:32:48 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:33:08 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:33:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:33:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:33:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:33:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:33:11 [WARNING ] (13, 1, 'bot')
2018-11-19 14:33:11 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:33:11 [DEBUG   ] Finished updating board flags in 1.543 seconds
2018-11-19 14:33:28 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:33:48 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:34:08 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 14:34:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:34:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:34:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:34:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:34:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:34:12 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:34:12 [DEBUG   ] Finished updating board flags in 2.199 seconds
2018-11-19 14:34:28 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 14:34:48 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:34:53 [DEBUG   ] Got command RPT from MCS: ref #559497
2018-11-19 14:34:53 [DEBUG   ] summary = ERROR
2018-11-19 14:34:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559497  13 58441 52493621 A  ERRORERROR'
2018-11-19 14:34:53 [DEBUG   ] Got command RPT from MCS: ref #559498
2018-11-19 14:34:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:34:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559498  77 58441 52493813 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:34:54 [DEBUG   ] Got command RPT from MCS: ref #559499
2018-11-19 14:34:54 [DEBUG   ] summary = ERROR
2018-11-19 14:34:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559499  13 58441 52494017 A  ERRORERROR'
2018-11-19 14:34:54 [DEBUG   ] Got command RPT from MCS: ref #559500
2018-11-19 14:34:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:34:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559500  77 58441 52494221 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:34:54 [DEBUG   ] Got command RPT from MCS: ref #559501
2018-11-19 14:34:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:34:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559501  12 58441 52494425 A  ERRORB˜™š'
2018-11-19 14:35:08 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:35:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 14:35:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:35:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:35:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:35:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:35:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:35:12 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:35:12 [DEBUG   ] Finished updating board flags in 2.274 seconds
2018-11-19 14:35:28 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:35:46 [DEBUG   ] Got command RPT from MCS: ref #559502
2018-11-19 14:35:46 [DEBUG   ] summary = ERROR
2018-11-19 14:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559502  13 58441 52546413 A  ERRORERROR'
2018-11-19 14:35:46 [DEBUG   ] Got command RPT from MCS: ref #559503
2018-11-19 14:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559503  77 58441 52546414 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:35:48 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:36:08 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 14:36:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:36:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:36:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:36:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:36:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:36:12 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:36:12 [DEBUG   ] Finished updating board flags in 1.852 seconds
2018-11-19 14:36:28 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:36:48 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:37:08 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:37:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:37:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:37:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:37:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:37:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:37:12 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:37:12 [DEBUG   ] Finished updating board flags in 1.824 seconds
2018-11-19 14:37:28 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 14:37:48 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:38:08 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:38:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 14:38:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:38:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:38:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:38:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:38:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:38:12 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:38:12 [DEBUG   ] Finished updating board flags in 1.780 seconds
2018-11-19 14:38:28 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:38:48 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 14:39:08 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 14:39:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:39:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 14:39:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:39:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:39:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:39:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:39:12 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:39:12 [DEBUG   ] Finished updating board flags in 1.810 seconds
2018-11-19 14:39:28 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 14:39:48 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 14:39:53 [DEBUG   ] Got command RPT from MCS: ref #559525
2018-11-19 14:39:53 [DEBUG   ] summary = ERROR
2018-11-19 14:39:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559525  13 58441 52793701 A  ERRORERROR'
2018-11-19 14:39:53 [DEBUG   ] Got command RPT from MCS: ref #559526
2018-11-19 14:39:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:39:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559526  77 58441 52793905 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:39:54 [DEBUG   ] Got command RPT from MCS: ref #559527
2018-11-19 14:39:54 [DEBUG   ] summary = ERROR
2018-11-19 14:39:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559527  13 58441 52794109 A  ERRORERROR'
2018-11-19 14:39:54 [DEBUG   ] Got command RPT from MCS: ref #559529
2018-11-19 14:39:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:39:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559529  77 58441 52794301 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:39:54 [DEBUG   ] Got command RPT from MCS: ref #559531
2018-11-19 14:39:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:39:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559531  12 58441 52794505 A  ERRORB˜™š'
2018-11-19 14:40:08 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 14:40:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:40:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:40:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:40:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:40:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:40:12 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:40:12 [DEBUG   ] Finished updating board flags in 1.588 seconds
2018-11-19 14:40:28 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 14:40:46 [DEBUG   ] Got command RPT from MCS: ref #559539
2018-11-19 14:40:46 [DEBUG   ] summary = ERROR
2018-11-19 14:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559539  13 58441 52846417 A  ERRORERROR'
2018-11-19 14:40:46 [DEBUG   ] Got command RPT from MCS: ref #559540
2018-11-19 14:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559540  77 58441 52846418 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:40:48 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 14:41:08 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:41:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:41:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:41:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:41:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:41:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:41:12 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:41:12 [DEBUG   ] Finished updating board flags in 1.403 seconds
2018-11-19 14:41:28 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 14:41:48 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:42:08 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:42:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:42:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:42:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:42:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:42:12 [WARNING ] (13, 1, 'bot')
2018-11-19 14:42:12 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:42:12 [DEBUG   ] Finished updating board flags in 1.568 seconds
2018-11-19 14:42:28 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-19 14:42:48 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 14:43:08 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-19 14:43:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 14:43:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:43:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:43:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:43:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:43:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:43:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:43:13 [DEBUG   ] Finished updating board flags in 1.591 seconds
2018-11-19 14:43:29 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-19 14:43:49 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 14:44:09 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:44:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:44:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:44:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:44:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:44:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:44:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:44:13 [DEBUG   ] Finished updating board flags in 2.163 seconds
2018-11-19 14:44:29 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:44:49 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:44:53 [DEBUG   ] Got command RPT from MCS: ref #559560
2018-11-19 14:44:53 [DEBUG   ] summary = ERROR
2018-11-19 14:44:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559560  13 58441 53093785 A  ERRORERROR'
2018-11-19 14:44:53 [DEBUG   ] Got command RPT from MCS: ref #559561
2018-11-19 14:44:53 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:44:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559561  77 58441 53093989 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:44:54 [DEBUG   ] Got command RPT from MCS: ref #559562
2018-11-19 14:44:54 [DEBUG   ] summary = ERROR
2018-11-19 14:44:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559562  13 58441 53094193 A  ERRORERROR'
2018-11-19 14:44:54 [DEBUG   ] Got command RPT from MCS: ref #559563
2018-11-19 14:44:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:44:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559563  77 58441 53094385 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:44:54 [DEBUG   ] Got command RPT from MCS: ref #559564
2018-11-19 14:44:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:44:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559564  12 58441 53094589 A  ERRORB™™š'
2018-11-19 14:45:09 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:45:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:45:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:45:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:45:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:45:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:45:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:45:13 [DEBUG   ] Finished updating board flags in 1.881 seconds
2018-11-19 14:45:29 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:45:46 [DEBUG   ] Got command RPT from MCS: ref #559565
2018-11-19 14:45:46 [DEBUG   ] summary = ERROR
2018-11-19 14:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559565  13 58441 53146409 A  ERRORERROR'
2018-11-19 14:45:46 [DEBUG   ] Got command RPT from MCS: ref #559566
2018-11-19 14:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559566  77 58441 53146410 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:45:49 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 14:46:09 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:46:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 14:46:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:46:13 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 14:46:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:46:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:46:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:46:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:46:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:46:13 [DEBUG   ] Finished updating board flags in 1.823 seconds
2018-11-19 14:46:29 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:46:49 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:47:09 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:47:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 14:47:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:47:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:47:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:47:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:47:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:47:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:47:13 [DEBUG   ] Finished updating board flags in 1.794 seconds
2018-11-19 14:47:29 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:47:49 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:48:09 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 14:48:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:48:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:48:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:48:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:48:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:48:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:48:13 [DEBUG   ] Finished updating board flags in 1.603 seconds
2018-11-19 14:48:29 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:48:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:49:09 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:49:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:49:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:49:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:49:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:49:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:49:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:49:13 [DEBUG   ] Finished updating board flags in 1.613 seconds
2018-11-19 14:49:29 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:49:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:49:53 [DEBUG   ] Got command RPT from MCS: ref #559592
2018-11-19 14:49:53 [DEBUG   ] summary = ERROR
2018-11-19 14:49:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559592  13 58441 53393873 A  ERRORERROR'
2018-11-19 14:49:54 [DEBUG   ] Got command RPT from MCS: ref #559593
2018-11-19 14:49:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:49:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559593  77 58441 53394077 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:49:54 [DEBUG   ] Got command RPT from MCS: ref #559594
2018-11-19 14:49:54 [DEBUG   ] summary = ERROR
2018-11-19 14:49:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559594  13 58441 53394281 A  ERRORERROR'
2018-11-19 14:49:54 [DEBUG   ] Got command RPT from MCS: ref #559596
2018-11-19 14:49:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:49:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559596  77 58441 53394473 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:49:54 [DEBUG   ] Got command RPT from MCS: ref #559597
2018-11-19 14:49:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:49:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559597  12 58441 53394677 A  ERRORB™™š'
2018-11-19 14:50:09 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:50:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:50:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:50:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:50:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:50:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:50:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:50:13 [DEBUG   ] Finished updating board flags in 1.701 seconds
2018-11-19 14:50:29 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:50:46 [DEBUG   ] Got command RPT from MCS: ref #559602
2018-11-19 14:50:46 [DEBUG   ] summary = ERROR
2018-11-19 14:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559602  13 58441 53446413 A  ERRORERROR'
2018-11-19 14:50:46 [DEBUG   ] Got command RPT from MCS: ref #559603
2018-11-19 14:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559603  77 58441 53446414 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:50:49 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:51:09 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:51:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:51:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:51:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:51:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:51:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:51:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:51:13 [DEBUG   ] Finished updating board flags in 1.485 seconds
2018-11-19 14:51:29 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 14:51:49 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:52:09 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 14:52:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:52:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:52:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:52:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:52:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:52:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:52:13 [DEBUG   ] Finished updating board flags in 1.484 seconds
2018-11-19 14:52:29 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 14:52:49 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 14:53:09 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 14:53:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:53:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:53:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:53:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:53:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:53:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:53:13 [DEBUG   ] Finished updating board flags in 1.487 seconds
2018-11-19 14:53:29 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 14:53:49 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:54:09 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 14:54:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:54:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:54:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:54:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:54:13 [WARNING ] (13, 1, 'bot')
2018-11-19 14:54:13 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:54:13 [DEBUG   ] Finished updating board flags in 1.493 seconds
2018-11-19 14:54:29 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 14:54:49 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 14:54:53 [DEBUG   ] Got command RPT from MCS: ref #559623
2018-11-19 14:54:53 [DEBUG   ] summary = ERROR
2018-11-19 14:54:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559623  13 58441 53693965 A  ERRORERROR'
2018-11-19 14:54:54 [DEBUG   ] Got command RPT from MCS: ref #559624
2018-11-19 14:54:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:54:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559624  77 58441 53694157 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:54:54 [DEBUG   ] Got command RPT from MCS: ref #559625
2018-11-19 14:54:54 [DEBUG   ] summary = ERROR
2018-11-19 14:54:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559625  13 58441 53694361 A  ERRORERROR'
2018-11-19 14:54:54 [DEBUG   ] Got command RPT from MCS: ref #559626
2018-11-19 14:54:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:54:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559626  77 58441 53694565 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:54:54 [DEBUG   ] Got command RPT from MCS: ref #559627
2018-11-19 14:54:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:54:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559627  12 58441 53694757 A  ERRORB˜™š'
2018-11-19 14:55:09 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-19 14:55:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:55:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 14:55:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:55:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:55:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:55:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:55:14 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:55:14 [DEBUG   ] Finished updating board flags in 2.163 seconds
2018-11-19 14:55:30 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-19 14:55:46 [DEBUG   ] Got command RPT from MCS: ref #559632
2018-11-19 14:55:46 [DEBUG   ] summary = ERROR
2018-11-19 14:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559632  13 58441 53746417 A  ERRORERROR'
2018-11-19 14:55:46 [DEBUG   ] Got command RPT from MCS: ref #559633
2018-11-19 14:55:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559633  77 58441 53746418 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:55:50 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 14:56:10 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:56:13 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 14:56:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:56:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:56:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:56:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:56:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:56:14 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:56:14 [DEBUG   ] Finished updating board flags in 1.990 seconds
2018-11-19 14:56:30 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:56:50 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:57:10 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 14:57:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:57:14 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 14:57:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:57:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:57:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:57:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:57:14 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:57:14 [DEBUG   ] Finished updating board flags in 1.816 seconds
2018-11-19 14:57:30 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:57:50 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:58:10 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:58:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:58:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 14:58:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:58:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:58:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:58:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:58:14 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:58:14 [DEBUG   ] Finished updating board flags in 1.781 seconds
2018-11-19 14:58:30 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 14:58:50 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 14:59:10 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 14:59:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 14:59:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:59:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:59:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:59:14 [WARNING ] (13, 1, 'bot')
2018-11-19 14:59:14 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 14:59:14 [DEBUG   ] Finished updating board flags in 1.623 seconds
2018-11-19 14:59:30 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 14:59:50 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-19 14:59:54 [DEBUG   ] Got command RPT from MCS: ref #559655
2018-11-19 14:59:54 [DEBUG   ] summary = ERROR
2018-11-19 14:59:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559655  13 58441 53994049 A  ERRORERROR'
2018-11-19 14:59:54 [DEBUG   ] Got command RPT from MCS: ref #559656
2018-11-19 14:59:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:59:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559656  77 58441 53994241 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:59:54 [DEBUG   ] Got command RPT from MCS: ref #559657
2018-11-19 14:59:54 [DEBUG   ] summary = ERROR
2018-11-19 14:59:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559657  13 58441 53994445 A  ERRORERROR'
2018-11-19 14:59:54 [DEBUG   ] Got command RPT from MCS: ref #559659
2018-11-19 14:59:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 14:59:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559659  77 58441 53994649 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 14:59:54 [DEBUG   ] Got command RPT from MCS: ref #559660
2018-11-19 14:59:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 14:59:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559660  12 58441 53994853 A  ERRORB™™š'
2018-11-19 15:00:10 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:00:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:00:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:00:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:00:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:00:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:00:14 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:00:14 [DEBUG   ] Finished updating board flags in 1.584 seconds
2018-11-19 15:00:30 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:00:46 [DEBUG   ] Got command RPT from MCS: ref #559665
2018-11-19 15:00:46 [DEBUG   ] summary = ERROR
2018-11-19 15:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559665  13 58441 54046421 A  ERRORERROR'
2018-11-19 15:00:46 [DEBUG   ] Got command RPT from MCS: ref #559666
2018-11-19 15:00:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559666  77 58441 54046422 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:00:50 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 15:01:10 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:01:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:01:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:01:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:01:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:01:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:01:14 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:01:14 [DEBUG   ] Finished updating board flags in 1.524 seconds
2018-11-19 15:01:30 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:01:50 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:02:10 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:02:13 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 15:02:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:02:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:02:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:02:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:02:14 [WARNING ] (13, 1, 'bot')
2018-11-19 15:02:14 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:02:14 [DEBUG   ] Finished updating board flags in 1.546 seconds
2018-11-19 15:02:30 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:02:50 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:03:10 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 15:03:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:03:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:03:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:03:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:03:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:03:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:03:15 [DEBUG   ] Finished updating board flags in 2.236 seconds
2018-11-19 15:03:30 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:03:50 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:04:10 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:04:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:04:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:04:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:04:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:04:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:04:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:04:15 [DEBUG   ] Finished updating board flags in 2.131 seconds
2018-11-19 15:04:30 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:04:50 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:04:54 [DEBUG   ] Got command RPT from MCS: ref #559690
2018-11-19 15:04:54 [DEBUG   ] summary = ERROR
2018-11-19 15:04:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559690  13 58441 54294137 A  ERRORERROR'
2018-11-19 15:04:54 [DEBUG   ] Got command RPT from MCS: ref #559691
2018-11-19 15:04:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:04:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559691  77 58441 54294329 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:04:54 [DEBUG   ] Got command RPT from MCS: ref #559692
2018-11-19 15:04:54 [DEBUG   ] summary = ERROR
2018-11-19 15:04:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559692  13 58441 54294533 A  ERRORERROR'
2018-11-19 15:04:54 [DEBUG   ] Got command RPT from MCS: ref #559693
2018-11-19 15:04:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:04:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559693  77 58441 54294737 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:04:54 [DEBUG   ] Got command RPT from MCS: ref #559694
2018-11-19 15:04:54 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:04:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559694  12 58441 54294941 A  ERRORB—™š'
2018-11-19 15:05:10 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:05:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:05:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:05:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:05:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:05:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:05:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:05:15 [DEBUG   ] Finished updating board flags in 1.987 seconds
2018-11-19 15:05:30 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:05:46 [DEBUG   ] Got command RPT from MCS: ref #559695
2018-11-19 15:05:46 [DEBUG   ] summary = ERROR
2018-11-19 15:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559695  13 58441 54346421 A  ERRORERROR'
2018-11-19 15:05:46 [DEBUG   ] Got command RPT from MCS: ref #559696
2018-11-19 15:05:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559696  77 58441 54346422 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:05:50 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:06:10 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 15:06:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:06:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:06:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:06:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:06:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:06:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:06:15 [DEBUG   ] Finished updating board flags in 1.814 seconds
2018-11-19 15:06:30 [DEBUG   ] Finished updating ant stats in 0.997 seconds
2018-11-19 15:06:50 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 15:07:10 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:07:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 15:07:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:07:15 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 15:07:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:07:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:07:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:07:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:07:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:07:15 [DEBUG   ] Finished updating board flags in 1.779 seconds
2018-11-19 15:07:30 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 15:07:50 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 15:08:10 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 15:08:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:08:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:08:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:08:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:08:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:08:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:08:15 [DEBUG   ] Finished updating board flags in 1.610 seconds
2018-11-19 15:08:30 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:08:50 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 15:09:10 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 15:09:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:09:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:09:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:09:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:09:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:09:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:09:15 [DEBUG   ] Finished updating board flags in 1.689 seconds
2018-11-19 15:09:30 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 15:09:50 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 15:09:54 [DEBUG   ] Got command RPT from MCS: ref #559718
2018-11-19 15:09:54 [DEBUG   ] summary = ERROR
2018-11-19 15:09:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559718  13 58441 54594221 A  ERRORERROR'
2018-11-19 15:09:54 [DEBUG   ] Got command RPT from MCS: ref #559719
2018-11-19 15:09:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:09:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559719  77 58441 54594413 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:09:54 [DEBUG   ] Got command RPT from MCS: ref #559720
2018-11-19 15:09:54 [DEBUG   ] summary = ERROR
2018-11-19 15:09:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559720  13 58441 54594617 A  ERRORERROR'
2018-11-19 15:09:54 [DEBUG   ] Got command RPT from MCS: ref #559722
2018-11-19 15:09:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:09:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559722  77 58441 54594821 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:09:55 [DEBUG   ] Got command RPT from MCS: ref #559723
2018-11-19 15:09:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:09:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559723  12 58441 54595013 A  ERRORB˜™š'
2018-11-19 15:10:10 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 15:10:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:10:15 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 15:10:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:10:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:10:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:10:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:10:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:10:15 [DEBUG   ] Finished updating board flags in 1.498 seconds
2018-11-19 15:10:30 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 15:10:46 [DEBUG   ] Got command RPT from MCS: ref #559732
2018-11-19 15:10:46 [DEBUG   ] summary = ERROR
2018-11-19 15:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559732  13 58441 54646421 A  ERRORERROR'
2018-11-19 15:10:46 [DEBUG   ] Got command RPT from MCS: ref #559733
2018-11-19 15:10:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559733  77 58441 54646422 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:10:50 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 15:11:11 [DEBUG   ] Finished updating ant stats in 1.396 seconds
2018-11-19 15:11:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:11:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:11:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:11:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:11:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:11:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:11:15 [DEBUG   ] Finished updating board flags in 1.468 seconds
2018-11-19 15:11:31 [DEBUG   ] Finished updating ant stats in 1.209 seconds
2018-11-19 15:11:51 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:12:11 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 15:12:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:12:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:12:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:12:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:12:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:12:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:12:15 [DEBUG   ] Finished updating board flags in 1.472 seconds
2018-11-19 15:12:31 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:12:51 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:13:11 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:13:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:13:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:13:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:13:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:13:15 [WARNING ] (13, 1, 'bot')
2018-11-19 15:13:15 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:13:15 [DEBUG   ] Finished updating board flags in 1.477 seconds
2018-11-19 15:13:31 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:13:51 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:14:11 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-19 15:14:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:14:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:14:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:14:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:14:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:14:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:14:16 [DEBUG   ] Finished updating board flags in 2.225 seconds
2018-11-19 15:14:31 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:14:51 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 15:14:54 [DEBUG   ] Got command RPT from MCS: ref #559753
2018-11-19 15:14:54 [DEBUG   ] summary = ERROR
2018-11-19 15:14:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559753  13 58441 54894301 A  ERRORERROR'
2018-11-19 15:14:54 [DEBUG   ] Got command RPT from MCS: ref #559754
2018-11-19 15:14:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:14:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559754  77 58441 54894505 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:14:54 [DEBUG   ] Got command RPT from MCS: ref #559755
2018-11-19 15:14:54 [DEBUG   ] summary = ERROR
2018-11-19 15:14:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559755  13 58441 54894697 A  ERRORERROR'
2018-11-19 15:14:54 [DEBUG   ] Got command RPT from MCS: ref #559756
2018-11-19 15:14:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:14:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559756  77 58441 54894901 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:14:55 [DEBUG   ] Got command RPT from MCS: ref #559757
2018-11-19 15:14:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:14:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559757  12 58441 54895105 A  ERRORB˜™š'
2018-11-19 15:15:11 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 15:15:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:15:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:15:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:15:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:15:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:15:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:15:16 [DEBUG   ] Finished updating board flags in 2.166 seconds
2018-11-19 15:15:31 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:15:46 [DEBUG   ] Got command RPT from MCS: ref #559758
2018-11-19 15:15:46 [DEBUG   ] summary = ERROR
2018-11-19 15:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559758  13 58441 54946421 A  ERRORERROR'
2018-11-19 15:15:46 [DEBUG   ] Got command RPT from MCS: ref #559759
2018-11-19 15:15:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559759  77 58441 54946422 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:15:51 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:16:11 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:16:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:16:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:16:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:16:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:16:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:16:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:16:16 [DEBUG   ] Finished updating board flags in 2.010 seconds
2018-11-19 15:16:31 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:16:51 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:17:11 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:17:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:17:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:17:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:17:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:17:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:17:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:17:16 [DEBUG   ] Finished updating board flags in 2.090 seconds
2018-11-19 15:17:31 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 15:17:51 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 15:18:11 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 15:18:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:18:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 15:18:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:18:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:18:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:18:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:18:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:18:16 [DEBUG   ] Finished updating board flags in 2.008 seconds
2018-11-19 15:18:31 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 15:18:51 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:19:11 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 15:19:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:19:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:19:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:19:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:19:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:19:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:19:16 [DEBUG   ] Finished updating board flags in 1.913 seconds
2018-11-19 15:19:31 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 15:19:51 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 15:19:54 [DEBUG   ] Got command RPT from MCS: ref #559785
2018-11-19 15:19:54 [DEBUG   ] summary = ERROR
2018-11-19 15:19:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559785  13 58441 55194385 A  ERRORERROR'
2018-11-19 15:19:54 [DEBUG   ] Got command RPT from MCS: ref #559786
2018-11-19 15:19:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:19:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559786  77 58441 55194589 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:19:54 [DEBUG   ] Got command RPT from MCS: ref #559787
2018-11-19 15:19:54 [DEBUG   ] summary = ERROR
2018-11-19 15:19:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559787  13 58441 55194793 A  ERRORERROR'
2018-11-19 15:19:54 [DEBUG   ] Got command RPT from MCS: ref #559789
2018-11-19 15:19:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:19:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559789  77 58441 55194985 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:19:55 [DEBUG   ] Got command RPT from MCS: ref #559790
2018-11-19 15:19:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:19:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559790  12 58441 55195189 A  ERRORB˜™š'
2018-11-19 15:20:11 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 15:20:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:20:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:20:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:20:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:20:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:20:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:20:16 [DEBUG   ] Finished updating board flags in 1.815 seconds
2018-11-19 15:20:31 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 15:20:46 [DEBUG   ] Got command RPT from MCS: ref #559795
2018-11-19 15:20:46 [DEBUG   ] summary = ERROR
2018-11-19 15:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559795  13 58441 55246421 A  ERRORERROR'
2018-11-19 15:20:46 [DEBUG   ] Got command RPT from MCS: ref #559796
2018-11-19 15:20:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559796  77 58441 55246422 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:20:51 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 15:21:11 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 15:21:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 15:21:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:21:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 15:21:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:21:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:21:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:21:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:21:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:21:16 [DEBUG   ] Finished updating board flags in 1.769 seconds
2018-11-19 15:21:31 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 15:21:51 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 15:22:11 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-19 15:22:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 15:22:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:22:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:22:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:22:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:22:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:22:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:22:16 [DEBUG   ] Finished updating board flags in 1.623 seconds
2018-11-19 15:22:31 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 15:22:51 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 15:23:11 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 15:23:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:23:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:23:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:23:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:23:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:23:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:23:16 [DEBUG   ] Finished updating board flags in 1.637 seconds
2018-11-19 15:23:31 [DEBUG   ] Finished updating ant stats in 0.592 seconds
2018-11-19 15:23:52 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-19 15:24:12 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-19 15:24:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 15:24:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:24:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:24:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:24:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:24:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:24:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:24:16 [DEBUG   ] Finished updating board flags in 1.615 seconds
2018-11-19 15:24:32 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:24:52 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:24:54 [DEBUG   ] Got command RPT from MCS: ref #559816
2018-11-19 15:24:54 [DEBUG   ] summary = ERROR
2018-11-19 15:24:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559816  13 58441 55494465 A  ERRORERROR'
2018-11-19 15:24:54 [DEBUG   ] Got command RPT from MCS: ref #559817
2018-11-19 15:24:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:24:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559817  77 58441 55494669 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:24:54 [DEBUG   ] Got command RPT from MCS: ref #559818
2018-11-19 15:24:54 [DEBUG   ] summary = ERROR
2018-11-19 15:24:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559818  13 58441 55494873 A  ERRORERROR'
2018-11-19 15:24:55 [DEBUG   ] Got command RPT from MCS: ref #559819
2018-11-19 15:24:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:24:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559819  77 58441 55495077 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:24:55 [DEBUG   ] Got command RPT from MCS: ref #559820
2018-11-19 15:24:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:24:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559820  12 58441 55495269 A  ERRORB™™š'
2018-11-19 15:25:12 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 15:25:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:25:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:25:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:25:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:25:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:25:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:25:16 [DEBUG   ] Finished updating board flags in 1.603 seconds
2018-11-19 15:25:32 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:25:46 [DEBUG   ] Got command RPT from MCS: ref #559825
2018-11-19 15:25:46 [DEBUG   ] summary = ERROR
2018-11-19 15:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559825  13 58441 55546425 A  ERRORERROR'
2018-11-19 15:25:46 [DEBUG   ] Got command RPT from MCS: ref #559826
2018-11-19 15:25:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559826  77 58441 55546426 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:25:52 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:26:12 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:26:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:26:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:26:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:26:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:26:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:26:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:26:16 [DEBUG   ] Finished updating board flags in 1.535 seconds
2018-11-19 15:26:32 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:26:52 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:27:12 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:27:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:27:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:27:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:27:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:27:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:27:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:27:16 [DEBUG   ] Finished updating board flags in 1.513 seconds
2018-11-19 15:27:32 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:27:52 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 15:28:12 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 15:28:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:28:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:28:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:28:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:28:16 [WARNING ] (13, 1, 'bot')
2018-11-19 15:28:16 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:28:16 [DEBUG   ] Finished updating board flags in 1.515 seconds
2018-11-19 15:28:32 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:28:52 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:29:12 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:29:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:29:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:29:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:29:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:29:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:29:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:29:17 [DEBUG   ] Finished updating board flags in 2.236 seconds
2018-11-19 15:29:32 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:29:52 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:29:54 [DEBUG   ] Got command RPT from MCS: ref #559848
2018-11-19 15:29:54 [DEBUG   ] summary = ERROR
2018-11-19 15:29:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559848  13 58441 55794561 A  ERRORERROR'
2018-11-19 15:29:54 [DEBUG   ] Got command RPT from MCS: ref #559849
2018-11-19 15:29:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:29:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559849  77 58441 55794753 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:29:54 [DEBUG   ] Got command RPT from MCS: ref #559850
2018-11-19 15:29:54 [DEBUG   ] summary = ERROR
2018-11-19 15:29:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559850  13 58441 55794957 A  ERRORERROR'
2018-11-19 15:29:55 [DEBUG   ] Got command RPT from MCS: ref #559852
2018-11-19 15:29:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:29:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559852  77 58441 55795161 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:29:55 [DEBUG   ] Got command RPT from MCS: ref #559854
2018-11-19 15:29:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:29:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559854  12 58441 55795365 A  ERRORB˜™š'
2018-11-19 15:30:12 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:30:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:30:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:30:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:30:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:30:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:30:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:30:17 [DEBUG   ] Finished updating board flags in 2.189 seconds
2018-11-19 15:30:32 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:30:46 [DEBUG   ] Got command RPT from MCS: ref #559858
2018-11-19 15:30:46 [DEBUG   ] summary = ERROR
2018-11-19 15:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559858  13 58441 55846429 A  ERRORERROR'
2018-11-19 15:30:46 [DEBUG   ] Got command RPT from MCS: ref #559859
2018-11-19 15:30:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559859  77 58441 55846430 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:30:52 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:31:12 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:31:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:31:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:31:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:31:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:31:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:31:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:31:17 [DEBUG   ] Finished updating board flags in 2.012 seconds
2018-11-19 15:31:32 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:31:52 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 15:32:12 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:32:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:32:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 15:32:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:32:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:32:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:32:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:32:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:32:17 [DEBUG   ] Finished updating board flags in 2.021 seconds
2018-11-19 15:32:32 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:32:52 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:33:12 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 15:33:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 15:33:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:33:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:33:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:33:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:33:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:33:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:33:17 [DEBUG   ] Finished updating board flags in 1.919 seconds
2018-11-19 15:33:32 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 15:33:52 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 15:34:12 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:34:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:34:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 15:34:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:34:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:34:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:34:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:34:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:34:17 [DEBUG   ] Finished updating board flags in 1.826 seconds
2018-11-19 15:34:32 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:34:52 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 15:34:54 [DEBUG   ] Got command RPT from MCS: ref #559883
2018-11-19 15:34:54 [DEBUG   ] summary = ERROR
2018-11-19 15:34:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559883  13 58441 56094641 A  ERRORERROR'
2018-11-19 15:34:54 [DEBUG   ] Got command RPT from MCS: ref #559884
2018-11-19 15:34:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:34:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559884  77 58441 56094845 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:34:55 [DEBUG   ] Got command RPT from MCS: ref #559885
2018-11-19 15:34:55 [DEBUG   ] summary = ERROR
2018-11-19 15:34:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559885  13 58441 56095037 A  ERRORERROR'
2018-11-19 15:34:55 [DEBUG   ] Got command RPT from MCS: ref #559886
2018-11-19 15:34:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:34:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559886  77 58441 56095241 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:34:55 [DEBUG   ] Got command RPT from MCS: ref #559887
2018-11-19 15:34:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:34:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559887  12 58441 56095445 A  ERRORB™™š'
2018-11-19 15:35:12 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 15:35:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:35:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:35:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:35:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:35:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:35:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:35:17 [DEBUG   ] Finished updating board flags in 1.794 seconds
2018-11-19 15:35:32 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 15:35:46 [DEBUG   ] Got command RPT from MCS: ref #559888
2018-11-19 15:35:46 [DEBUG   ] summary = ERROR
2018-11-19 15:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559888  13 58441 56146429 A  ERRORERROR'
2018-11-19 15:35:46 [DEBUG   ] Got command RPT from MCS: ref #559889
2018-11-19 15:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559889  77 58441 56146430 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:35:52 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 15:36:12 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:36:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:36:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:36:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:36:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:36:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:36:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:36:17 [DEBUG   ] Finished updating board flags in 1.664 seconds
2018-11-19 15:36:32 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 15:36:52 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:37:12 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:37:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:37:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:37:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:37:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:37:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:37:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:37:17 [DEBUG   ] Finished updating board flags in 1.527 seconds
2018-11-19 15:37:32 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 15:37:52 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 15:38:12 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 15:38:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:38:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:38:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:38:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:38:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:38:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:38:17 [DEBUG   ] Finished updating board flags in 1.466 seconds
2018-11-19 15:38:32 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 15:38:53 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-19 15:39:13 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 15:39:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:39:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:39:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:39:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:39:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:39:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:39:17 [DEBUG   ] Finished updating board flags in 1.466 seconds
2018-11-19 15:39:33 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:39:53 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:39:54 [DEBUG   ] Got command RPT from MCS: ref #559911
2018-11-19 15:39:54 [DEBUG   ] summary = ERROR
2018-11-19 15:39:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559911  13 58441 56394729 A  ERRORERROR'
2018-11-19 15:39:54 [DEBUG   ] Got command RPT from MCS: ref #559912
2018-11-19 15:39:54 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:39:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559912  77 58441 56394933 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:39:55 [DEBUG   ] Got command RPT from MCS: ref #559913
2018-11-19 15:39:55 [DEBUG   ] summary = ERROR
2018-11-19 15:39:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559913  13 58441 56395137 A  ERRORERROR'
2018-11-19 15:39:55 [DEBUG   ] Got command RPT from MCS: ref #559915
2018-11-19 15:39:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:39:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559915  77 58441 56395329 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:39:55 [DEBUG   ] Got command RPT from MCS: ref #559916
2018-11-19 15:39:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:39:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559916  12 58441 56395533 A  ERRORB˜™š'
2018-11-19 15:40:13 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:40:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:40:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:40:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:40:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:40:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:40:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:40:17 [DEBUG   ] Finished updating board flags in 1.483 seconds
2018-11-19 15:40:33 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:40:46 [DEBUG   ] Got command RPT from MCS: ref #559925
2018-11-19 15:40:46 [DEBUG   ] summary = ERROR
2018-11-19 15:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559925  13 58441 56446429 A  ERRORERROR'
2018-11-19 15:40:46 [DEBUG   ] Got command RPT from MCS: ref #559926
2018-11-19 15:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559926  77 58441 56446430 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:40:53 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:41:13 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:41:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 15:41:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:41:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:41:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:41:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:41:17 [WARNING ] (13, 1, 'bot')
2018-11-19 15:41:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:41:17 [DEBUG   ] Finished updating board flags in 1.517 seconds
2018-11-19 15:41:33 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:41:53 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 15:42:13 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 15:42:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 15:42:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:42:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:42:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:42:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:42:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:42:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:42:18 [DEBUG   ] Finished updating board flags in 2.141 seconds
2018-11-19 15:42:33 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:42:53 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:43:13 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:43:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:43:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 15:43:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:43:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:43:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:43:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:43:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:43:18 [DEBUG   ] Finished updating board flags in 1.954 seconds
2018-11-19 15:43:33 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:43:53 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:44:13 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:44:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:44:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:44:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:44:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:44:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:44:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:44:18 [DEBUG   ] Finished updating board flags in 1.825 seconds
2018-11-19 15:44:33 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:44:53 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:44:54 [DEBUG   ] Got command RPT from MCS: ref #559946
2018-11-19 15:44:54 [DEBUG   ] summary = ERROR
2018-11-19 15:44:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559946  13 58441 56694817 A  ERRORERROR'
2018-11-19 15:44:55 [DEBUG   ] Got command RPT from MCS: ref #559947
2018-11-19 15:44:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:44:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559947  77 58441 56695009 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:44:55 [DEBUG   ] Got command RPT from MCS: ref #559948
2018-11-19 15:44:55 [DEBUG   ] summary = ERROR
2018-11-19 15:44:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559948  13 58441 56695213 A  ERRORERROR'
2018-11-19 15:44:55 [DEBUG   ] Got command RPT from MCS: ref #559949
2018-11-19 15:44:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:44:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559949  77 58441 56695417 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:44:55 [DEBUG   ] Got command RPT from MCS: ref #559950
2018-11-19 15:44:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:44:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559950  12 58441 56695621 A  ERRORB—™š'
2018-11-19 15:45:13 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:45:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 15:45:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:45:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 15:45:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:45:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:45:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:45:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:45:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:45:18 [DEBUG   ] Finished updating board flags in 1.803 seconds
2018-11-19 15:45:33 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:45:46 [DEBUG   ] Got command RPT from MCS: ref #559951
2018-11-19 15:45:46 [DEBUG   ] summary = ERROR
2018-11-19 15:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559951  13 58441 56746429 A  ERRORERROR'
2018-11-19 15:45:46 [DEBUG   ] Got command RPT from MCS: ref #559952
2018-11-19 15:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559952  77 58441 56746430 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:45:53 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:46:13 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:46:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 15:46:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:46:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa7, chip 5, bottom port
2018-11-19 15:46:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:46:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:46:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:46:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:46:18 [WARNING ] (25, 5, 'bot')
2018-11-19 15:46:18 [WARNING ] (25, 5, 'bot')
2018-11-19 15:46:18 [WARNING ] (25, 5, 'bot')
2018-11-19 15:46:18 [WARNING ] (25, 5, 'bot')
2018-11-19 15:46:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:46:18 [DEBUG   ] Finished updating board flags in 1.795 seconds
2018-11-19 15:46:33 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:46:53 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:47:13 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:47:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:47:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:47:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:47:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:47:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:47:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:47:18 [DEBUG   ] Finished updating board flags in 1.642 seconds
2018-11-19 15:47:33 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 15:47:53 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 15:48:13 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 15:48:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:48:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:48:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:48:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:48:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:48:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:48:18 [DEBUG   ] Finished updating board flags in 1.765 seconds
2018-11-19 15:48:33 [DEBUG   ] Finished updating ant stats in 0.997 seconds
2018-11-19 15:48:53 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:49:13 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:49:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:49:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:49:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:49:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:49:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:49:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:49:18 [DEBUG   ] Finished updating board flags in 1.369 seconds
2018-11-19 15:49:33 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 15:49:53 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 15:49:54 [DEBUG   ] Got command RPT from MCS: ref #559978
2018-11-19 15:49:54 [DEBUG   ] summary = ERROR
2018-11-19 15:49:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559978  13 58441 56994894 A  ERRORERROR'
2018-11-19 15:49:55 [DEBUG   ] Got command RPT from MCS: ref #559979
2018-11-19 15:49:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:49:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559979  77 58441 56995097 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:49:55 [DEBUG   ] Got command RPT from MCS: ref #559980
2018-11-19 15:49:55 [DEBUG   ] summary = ERROR
2018-11-19 15:49:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559980  13 58441 56995301 A  ERRORERROR'
2018-11-19 15:49:55 [DEBUG   ] Got command RPT from MCS: ref #559982
2018-11-19 15:49:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:49:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559982  77 58441 56995505 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:49:55 [DEBUG   ] Got command RPT from MCS: ref #559984
2018-11-19 15:49:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:49:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559984  12 58441 56995697 A  ERRORB˜™š'
2018-11-19 15:50:13 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 15:50:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:50:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:50:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:50:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:50:18 [WARNING ] (13, 1, 'bot')
2018-11-19 15:50:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:50:18 [DEBUG   ] Finished updating board flags in 1.379 seconds
2018-11-19 15:50:33 [DEBUG   ] Finished updating ant stats in 0.794 seconds
2018-11-19 15:50:46 [DEBUG   ] Got command RPT from MCS: ref #559988
2018-11-19 15:50:46 [DEBUG   ] summary = ERROR
2018-11-19 15:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559988  13 58441 57046437 A  ERRORERROR'
2018-11-19 15:50:46 [DEBUG   ] Got command RPT from MCS: ref #559989
2018-11-19 15:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   559989  77 58441 57046438 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:50:53 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 15:51:13 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 15:51:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:51:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:51:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:51:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:51:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:51:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:51:19 [DEBUG   ] Finished updating board flags in 2.050 seconds
2018-11-19 15:51:33 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 15:51:53 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 15:52:14 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-19 15:52:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:52:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:52:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:52:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:52:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:52:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:52:19 [DEBUG   ] Finished updating board flags in 1.974 seconds
2018-11-19 15:52:34 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 15:52:54 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:53:14 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:53:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:53:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:53:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:53:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:53:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:53:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:53:19 [DEBUG   ] Finished updating board flags in 1.820 seconds
2018-11-19 15:53:34 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:53:54 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:54:14 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-19 15:54:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:54:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:54:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:54:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:54:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:54:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:54:19 [DEBUG   ] Finished updating board flags in 1.818 seconds
2018-11-19 15:54:34 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:54:54 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:54:54 [DEBUG   ] Got command RPT from MCS: ref #560009
2018-11-19 15:54:54 [DEBUG   ] summary = ERROR
2018-11-19 15:54:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560009  13 58441 57294985 A  ERRORERROR'
2018-11-19 15:54:55 [DEBUG   ] Got command RPT from MCS: ref #560010
2018-11-19 15:54:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:54:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560010  77 58441 57295189 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:54:55 [DEBUG   ] Got command RPT from MCS: ref #560011
2018-11-19 15:54:55 [DEBUG   ] summary = ERROR
2018-11-19 15:54:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560011  13 58441 57295381 A  ERRORERROR'
2018-11-19 15:54:55 [DEBUG   ] Got command RPT from MCS: ref #560012
2018-11-19 15:54:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:54:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560012  77 58441 57295585 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:54:55 [DEBUG   ] Got command RPT from MCS: ref #560013
2018-11-19 15:54:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:54:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560013  12 58441 57295789 A  ERRORB˜™š'
2018-11-19 15:55:14 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:55:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:55:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:55:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:55:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:55:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:55:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:55:19 [DEBUG   ] Finished updating board flags in 1.824 seconds
2018-11-19 15:55:34 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:55:46 [DEBUG   ] Got command RPT from MCS: ref #560018
2018-11-19 15:55:46 [DEBUG   ] summary = ERROR
2018-11-19 15:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560018  13 58441 57346433 A  ERRORERROR'
2018-11-19 15:55:46 [DEBUG   ] Got command RPT from MCS: ref #560019
2018-11-19 15:55:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560019  77 58441 57346434 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:55:54 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:56:14 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:56:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 15:56:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:56:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:56:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:56:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:56:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:56:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:56:19 [DEBUG   ] Finished updating board flags in 1.764 seconds
2018-11-19 15:56:34 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:56:54 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:57:14 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:57:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:57:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:57:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:57:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:57:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:57:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:57:19 [DEBUG   ] Finished updating board flags in 1.652 seconds
2018-11-19 15:57:34 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:57:54 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:58:14 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:58:18 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 15:58:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:58:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 15:58:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 15:58:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:58:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:58:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:58:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:58:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:58:19 [DEBUG   ] Finished updating board flags in 1.543 seconds
2018-11-19 15:58:34 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:58:54 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 15:59:14 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 15:59:18 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 15:59:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 15:59:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:59:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:59:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:59:19 [WARNING ] (13, 1, 'bot')
2018-11-19 15:59:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 15:59:19 [DEBUG   ] Finished updating board flags in 1.518 seconds
2018-11-19 15:59:34 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:59:54 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 15:59:55 [DEBUG   ] Got command RPT from MCS: ref #560041
2018-11-19 15:59:55 [DEBUG   ] summary = ERROR
2018-11-19 15:59:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560041  13 58441 57595073 A  ERRORERROR'
2018-11-19 15:59:55 [DEBUG   ] Got command RPT from MCS: ref #560042
2018-11-19 15:59:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:59:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560042  77 58441 57595265 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:59:55 [DEBUG   ] Got command RPT from MCS: ref #560043
2018-11-19 15:59:55 [DEBUG   ] summary = ERROR
2018-11-19 15:59:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560043  13 58441 57595469 A  ERRORERROR'
2018-11-19 15:59:55 [DEBUG   ] Got command RPT from MCS: ref #560045
2018-11-19 15:59:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 15:59:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560045  77 58441 57595673 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 15:59:55 [DEBUG   ] Got command RPT from MCS: ref #560047
2018-11-19 15:59:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 15:59:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560047  12 58441 57595865 A  ERRORB˜™š'
2018-11-19 16:00:14 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 16:00:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:00:19 [WARNING ] (13, 1, 'bot')
2018-11-19 16:00:19 [WARNING ] (13, 1, 'bot')
2018-11-19 16:00:19 [WARNING ] (13, 1, 'bot')
2018-11-19 16:00:19 [WARNING ] (13, 1, 'bot')
2018-11-19 16:00:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:00:19 [DEBUG   ] Finished updating board flags in 1.558 seconds
2018-11-19 16:00:34 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 16:00:46 [DEBUG   ] Got command RPT from MCS: ref #560051
2018-11-19 16:00:46 [DEBUG   ] summary = ERROR
2018-11-19 16:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560051  13 58441 57646437 A  ERRORERROR'
2018-11-19 16:00:46 [DEBUG   ] Got command RPT from MCS: ref #560052
2018-11-19 16:00:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560052  77 58441 57646438 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:00:54 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-19 16:01:14 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 16:01:19 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 16:01:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:01:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 16:01:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:01:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:01:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:01:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:01:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:01:20 [DEBUG   ] Finished updating board flags in 2.197 seconds
2018-11-19 16:01:34 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 16:01:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:02:14 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:02:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 16:02:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:02:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 16:02:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:02:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:02:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:02:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:02:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:02:20 [DEBUG   ] Finished updating board flags in 1.927 seconds
2018-11-19 16:02:34 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:02:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:03:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:03:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:03:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:03:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:03:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:03:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:03:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:03:20 [DEBUG   ] Finished updating board flags in 1.825 seconds
2018-11-19 16:03:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:03:54 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:04:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:04:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:04:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 16:04:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:04:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:04:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:04:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:04:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:04:20 [DEBUG   ] Finished updating board flags in 1.844 seconds
2018-11-19 16:04:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:04:54 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:04:55 [DEBUG   ] Got command RPT from MCS: ref #560076
2018-11-19 16:04:55 [DEBUG   ] summary = ERROR
2018-11-19 16:04:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560076  13 58441 57895161 A  ERRORERROR'
2018-11-19 16:04:55 [DEBUG   ] Got command RPT from MCS: ref #560077
2018-11-19 16:04:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:04:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560077  77 58441 57895353 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:04:55 [DEBUG   ] Got command RPT from MCS: ref #560078
2018-11-19 16:04:55 [DEBUG   ] summary = ERROR
2018-11-19 16:04:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560078  13 58441 57895557 A  ERRORERROR'
2018-11-19 16:04:55 [DEBUG   ] Got command RPT from MCS: ref #560079
2018-11-19 16:04:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:04:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560079  77 58441 57895761 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:04:55 [DEBUG   ] Got command RPT from MCS: ref #560080
2018-11-19 16:04:55 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:04:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560080  12 58441 57895953 A  ERRORB˜™š'
2018-11-19 16:05:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:05:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:05:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:05:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:05:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:05:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:05:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:05:20 [DEBUG   ] Finished updating board flags in 1.811 seconds
2018-11-19 16:05:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:05:46 [DEBUG   ] Got command RPT from MCS: ref #560081
2018-11-19 16:05:46 [DEBUG   ] summary = ERROR
2018-11-19 16:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560081  13 58441 57946441 A  ERRORERROR'
2018-11-19 16:05:46 [DEBUG   ] Got command RPT from MCS: ref #560082
2018-11-19 16:05:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560082  77 58441 57946442 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:05:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:06:14 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:06:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:06:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:06:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:06:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:06:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:06:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:06:20 [DEBUG   ] Finished updating board flags in 1.752 seconds
2018-11-19 16:06:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:06:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:07:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:07:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:07:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:07:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:07:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:07:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:07:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:07:20 [DEBUG   ] Finished updating board flags in 1.642 seconds
2018-11-19 16:07:34 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:07:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:08:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:08:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:08:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:08:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:08:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:08:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:08:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:08:20 [DEBUG   ] Finished updating board flags in 1.531 seconds
2018-11-19 16:08:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:08:54 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:09:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:09:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 16:09:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:09:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:09:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:09:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:09:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:09:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:09:20 [DEBUG   ] Finished updating board flags in 1.484 seconds
2018-11-19 16:09:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:09:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:09:55 [DEBUG   ] Got command RPT from MCS: ref #560104
2018-11-19 16:09:55 [DEBUG   ] summary = ERROR
2018-11-19 16:09:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560104  13 58441 58195245 A  ERRORERROR'
2018-11-19 16:09:55 [DEBUG   ] Got command RPT from MCS: ref #560105
2018-11-19 16:09:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:09:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560105  77 58441 58195437 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:09:55 [DEBUG   ] Got command RPT from MCS: ref #560106
2018-11-19 16:09:55 [DEBUG   ] summary = ERROR
2018-11-19 16:09:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560106  13 58441 58195641 A  ERRORERROR'
2018-11-19 16:09:55 [DEBUG   ] Got command RPT from MCS: ref #560108
2018-11-19 16:09:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:09:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560108  77 58441 58195845 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:09:56 [DEBUG   ] Got command RPT from MCS: ref #560109
2018-11-19 16:09:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:09:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560109  12 58441 58196049 A  ERRORB˜™š'
2018-11-19 16:10:14 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:10:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 16:10:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:10:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:10:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:10:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:10:20 [WARNING ] (13, 1, 'bot')
2018-11-19 16:10:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:10:20 [DEBUG   ] Finished updating board flags in 1.550 seconds
2018-11-19 16:10:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:10:46 [DEBUG   ] Got command RPT from MCS: ref #560118
2018-11-19 16:10:46 [DEBUG   ] summary = ERROR
2018-11-19 16:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560118  13 58441 58246433 A  ERRORERROR'
2018-11-19 16:10:46 [DEBUG   ] Got command RPT from MCS: ref #560119
2018-11-19 16:10:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560119  77 58441 58246434 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:10:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:11:14 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 16:11:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:11:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa7, chip 5, bottom port
2018-11-19 16:11:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:11:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:11:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:11:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:11:21 [WARNING ] (25, 5, 'bot')
2018-11-19 16:11:21 [WARNING ] (25, 5, 'bot')
2018-11-19 16:11:21 [WARNING ] (25, 5, 'bot')
2018-11-19 16:11:21 [WARNING ] (25, 5, 'bot')
2018-11-19 16:11:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:11:21 [DEBUG   ] Finished updating board flags in 2.258 seconds
2018-11-19 16:11:34 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-19 16:11:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:12:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:12:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:12:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:12:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:12:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:12:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:12:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:12:21 [DEBUG   ] Finished updating board flags in 2.188 seconds
2018-11-19 16:12:34 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 16:12:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:13:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:13:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:13:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:13:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:13:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:13:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:13:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:13:21 [DEBUG   ] Finished updating board flags in 1.959 seconds
2018-11-19 16:13:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:13:54 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 16:14:14 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 16:14:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:14:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:14:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:14:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:14:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:14:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:14:21 [DEBUG   ] Finished updating board flags in 1.783 seconds
2018-11-19 16:14:34 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 16:14:54 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 16:14:55 [DEBUG   ] Got command RPT from MCS: ref #560139
2018-11-19 16:14:55 [DEBUG   ] summary = ERROR
2018-11-19 16:14:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560139  13 58441 58495321 A  ERRORERROR'
2018-11-19 16:14:55 [DEBUG   ] Got command RPT from MCS: ref #560140
2018-11-19 16:14:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:14:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560140  77 58441 58495525 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:14:55 [DEBUG   ] Got command RPT from MCS: ref #560141
2018-11-19 16:14:55 [DEBUG   ] summary = ERROR
2018-11-19 16:14:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560141  13 58441 58495729 A  ERRORERROR'
2018-11-19 16:14:55 [DEBUG   ] Got command RPT from MCS: ref #560142
2018-11-19 16:14:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:14:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560142  77 58441 58495933 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:14:56 [DEBUG   ] Got command RPT from MCS: ref #560143
2018-11-19 16:14:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:14:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560143  12 58441 58496125 A  ERRORB˜™š'
2018-11-19 16:15:14 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 16:15:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:15:21 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 16:15:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:15:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:15:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:15:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:15:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:15:21 [DEBUG   ] Finished updating board flags in 1.623 seconds
2018-11-19 16:15:35 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-19 16:15:46 [DEBUG   ] Got command RPT from MCS: ref #560144
2018-11-19 16:15:46 [DEBUG   ] summary = ERROR
2018-11-19 16:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560144  13 58441 58546441 A  ERRORERROR'
2018-11-19 16:15:46 [DEBUG   ] Got command RPT from MCS: ref #560145
2018-11-19 16:15:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560145  77 58441 58546442 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:15:55 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 16:16:15 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 16:16:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:16:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:16:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:16:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:16:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:16:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:16:21 [DEBUG   ] Finished updating board flags in 1.585 seconds
2018-11-19 16:16:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 16:16:55 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 16:17:15 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 16:17:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:17:21 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 16:17:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 16:17:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 16:17:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:17:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:17:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:17:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:17:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:17:21 [DEBUG   ] Finished updating board flags in 1.414 seconds
2018-11-19 16:17:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 16:17:55 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:18:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 16:18:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:18:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:18:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:18:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:18:21 [WARNING ] (13, 1, 'bot')
2018-11-19 16:18:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:18:21 [DEBUG   ] Finished updating board flags in 1.397 seconds
2018-11-19 16:18:35 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-19 16:18:55 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 16:19:15 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 16:19:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:19:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:19:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:19:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:19:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:19:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:19:22 [DEBUG   ] Finished updating board flags in 2.217 seconds
2018-11-19 16:19:35 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 16:19:55 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 16:19:55 [DEBUG   ] Got command RPT from MCS: ref #560171
2018-11-19 16:19:55 [DEBUG   ] summary = ERROR
2018-11-19 16:19:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560171  13 58441 58795405 A  ERRORERROR'
2018-11-19 16:19:55 [DEBUG   ] Got command RPT from MCS: ref #560172
2018-11-19 16:19:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:19:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560172  77 58441 58795609 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:19:55 [DEBUG   ] Got command RPT from MCS: ref #560173
2018-11-19 16:19:55 [DEBUG   ] summary = ERROR
2018-11-19 16:19:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560173  13 58441 58795813 A  ERRORERROR'
2018-11-19 16:19:56 [DEBUG   ] Got command RPT from MCS: ref #560175
2018-11-19 16:19:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:19:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560175  77 58441 58796017 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:19:56 [DEBUG   ] Got command RPT from MCS: ref #560176
2018-11-19 16:19:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:19:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560176  12 58441 58796209 A  ERRORB™™š'
2018-11-19 16:20:15 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 16:20:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:20:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:20:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:20:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:20:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:20:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:20:22 [DEBUG   ] Finished updating board flags in 2.226 seconds
2018-11-19 16:20:35 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 16:20:46 [DEBUG   ] Got command RPT from MCS: ref #560181
2018-11-19 16:20:46 [DEBUG   ] summary = ERROR
2018-11-19 16:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560181  13 58441 58846437 A  ERRORERROR'
2018-11-19 16:20:46 [DEBUG   ] Got command RPT from MCS: ref #560182
2018-11-19 16:20:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560182  77 58441 58846438 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:20:55 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 16:21:15 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 16:21:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:21:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 16:21:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 16:21:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:21:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:21:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:21:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:21:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:21:22 [DEBUG   ] Finished updating board flags in 2.145 seconds
2018-11-19 16:21:35 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 16:21:55 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 16:22:15 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 16:22:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:22:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:22:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:22:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:22:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:22:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:22:22 [DEBUG   ] Finished updating board flags in 2.074 seconds
2018-11-19 16:22:35 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 16:22:55 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:23:15 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-19 16:23:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:23:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 16:23:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:23:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:23:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:23:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:23:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:23:22 [DEBUG   ] Finished updating board flags in 1.932 seconds
2018-11-19 16:23:35 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:23:55 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:24:15 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:24:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:24:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:24:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:24:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:24:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:24:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:24:22 [DEBUG   ] Finished updating board flags in 1.837 seconds
2018-11-19 16:24:35 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:24:55 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:24:55 [DEBUG   ] Got command RPT from MCS: ref #560202
2018-11-19 16:24:55 [DEBUG   ] summary = ERROR
2018-11-19 16:24:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560202  13 58441 59095489 A  ERRORERROR'
2018-11-19 16:24:55 [DEBUG   ] Got command RPT from MCS: ref #560203
2018-11-19 16:24:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:24:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560203  77 58441 59095693 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:24:55 [DEBUG   ] Got command RPT from MCS: ref #560204
2018-11-19 16:24:55 [DEBUG   ] summary = ERROR
2018-11-19 16:24:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560204  13 58441 59095897 A  ERRORERROR'
2018-11-19 16:24:56 [DEBUG   ] Got command RPT from MCS: ref #560205
2018-11-19 16:24:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:24:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560205  77 58441 59096101 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:24:56 [DEBUG   ] Got command RPT from MCS: ref #560206
2018-11-19 16:24:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:24:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560206  12 58441 59096293 A  ERRORB˜™š'
2018-11-19 16:25:15 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:25:21 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 16:25:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:25:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:25:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:25:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:25:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:25:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:25:22 [DEBUG   ] Finished updating board flags in 1.780 seconds
2018-11-19 16:25:35 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:25:46 [DEBUG   ] Got command RPT from MCS: ref #560211
2018-11-19 16:25:46 [DEBUG   ] summary = ERROR
2018-11-19 16:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560211  13 58441 59146441 A  ERRORERROR'
2018-11-19 16:25:46 [DEBUG   ] Got command RPT from MCS: ref #560212
2018-11-19 16:25:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560212  77 58441 59146442 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:25:55 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:26:15 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:26:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:26:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 16:26:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:26:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:26:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:26:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:26:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:26:22 [DEBUG   ] Finished updating board flags in 1.608 seconds
2018-11-19 16:26:35 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:26:55 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:27:15 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:27:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:27:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:27:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:27:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:27:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:27:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:27:22 [DEBUG   ] Finished updating board flags in 1.647 seconds
2018-11-19 16:27:35 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:27:55 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 16:28:15 [DEBUG   ] Finished updating ant stats in 0.797 seconds
2018-11-19 16:28:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:28:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:28:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:28:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:28:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:28:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:28:22 [DEBUG   ] Finished updating board flags in 1.615 seconds
2018-11-19 16:28:35 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-19 16:28:55 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 16:29:16 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-19 16:29:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:29:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:29:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:29:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:29:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:29:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:29:22 [DEBUG   ] Finished updating board flags in 1.518 seconds
2018-11-19 16:29:36 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 16:29:55 [DEBUG   ] Got command RPT from MCS: ref #560234
2018-11-19 16:29:55 [DEBUG   ] summary = ERROR
2018-11-19 16:29:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560234  13 58441 59395581 A  ERRORERROR'
2018-11-19 16:29:55 [DEBUG   ] Got command RPT from MCS: ref #560235
2018-11-19 16:29:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:29:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560235  77 58441 59395773 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:29:55 [DEBUG   ] Got command RPT from MCS: ref #560236
2018-11-19 16:29:55 [DEBUG   ] summary = ERROR
2018-11-19 16:29:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560236  13 58441 59395977 A  ERRORERROR'
2018-11-19 16:29:56 [DEBUG   ] Got command RPT from MCS: ref #560238
2018-11-19 16:29:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:29:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560238  77 58441 59396181 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:29:56 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 16:29:56 [DEBUG   ] Got command RPT from MCS: ref #560240
2018-11-19 16:29:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:29:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560240  12 58441 59396385 A  ERRORB™™š'
2018-11-19 16:30:16 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 16:30:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:30:22 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 16:30:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:30:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:30:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:30:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:30:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:30:22 [DEBUG   ] Finished updating board flags in 1.453 seconds
2018-11-19 16:30:36 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 16:30:46 [DEBUG   ] Got command RPT from MCS: ref #560244
2018-11-19 16:30:46 [DEBUG   ] summary = ERROR
2018-11-19 16:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560244  13 58441 59446437 A  ERRORERROR'
2018-11-19 16:30:46 [DEBUG   ] Got command RPT from MCS: ref #560245
2018-11-19 16:30:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560245  77 58441 59446438 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:30:56 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:31:16 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:31:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:31:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:31:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:31:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:31:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:31:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:31:22 [DEBUG   ] Finished updating board flags in 1.516 seconds
2018-11-19 16:31:36 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 16:31:56 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 16:32:16 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:32:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:32:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:32:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:32:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:32:22 [WARNING ] (13, 1, 'bot')
2018-11-19 16:32:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:32:22 [DEBUG   ] Finished updating board flags in 1.540 seconds
2018-11-19 16:32:36 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:32:56 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 16:33:16 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 16:33:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:33:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:33:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:33:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:33:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:33:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:33:23 [DEBUG   ] Finished updating board flags in 2.168 seconds
2018-11-19 16:33:36 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:33:56 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:34:16 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 16:34:22 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 16:34:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:34:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:34:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:34:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:34:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:34:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:34:23 [DEBUG   ] Finished updating board flags in 1.948 seconds
2018-11-19 16:34:36 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 16:34:55 [DEBUG   ] Got command RPT from MCS: ref #560269
2018-11-19 16:34:55 [DEBUG   ] summary = ERROR
2018-11-19 16:34:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560269  13 58441 59695661 A  ERRORERROR'
2018-11-19 16:34:55 [DEBUG   ] Got command RPT from MCS: ref #560270
2018-11-19 16:34:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:34:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560270  77 58441 59695865 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:34:56 [DEBUG   ] Got command RPT from MCS: ref #560271
2018-11-19 16:34:56 [DEBUG   ] summary = ERROR
2018-11-19 16:34:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560271  13 58441 59696069 A  ERRORERROR'
2018-11-19 16:34:56 [DEBUG   ] Got command RPT from MCS: ref #560272
2018-11-19 16:34:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:34:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560272  77 58441 59696265 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:34:56 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:34:56 [DEBUG   ] Got command RPT from MCS: ref #560273
2018-11-19 16:34:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:34:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560273  12 58441 59696469 A  ERRORB™™š'
2018-11-19 16:35:16 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:35:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:35:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 16:35:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:35:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:35:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:35:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:35:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:35:23 [DEBUG   ] Finished updating board flags in 1.877 seconds
2018-11-19 16:35:36 [DEBUG   ] Finished updating ant stats in 1.195 seconds
2018-11-19 16:35:46 [DEBUG   ] Got command RPT from MCS: ref #560274
2018-11-19 16:35:46 [DEBUG   ] summary = ERROR
2018-11-19 16:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560274  13 58441 59746445 A  ERRORERROR'
2018-11-19 16:35:46 [DEBUG   ] Got command RPT from MCS: ref #560275
2018-11-19 16:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560275  77 58441 59746446 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:35:56 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 16:36:16 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 16:36:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:36:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 16:36:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:36:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:36:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:36:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:36:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:36:23 [DEBUG   ] Finished updating board flags in 1.712 seconds
2018-11-19 16:36:36 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 16:36:56 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 16:37:16 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 16:37:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:37:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:37:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:37:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:37:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:37:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:37:23 [DEBUG   ] Finished updating board flags in 1.624 seconds
2018-11-19 16:37:36 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 16:37:56 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 16:38:16 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 16:38:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:38:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa7, chip 5, bottom port
2018-11-19 16:38:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:38:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:38:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:38:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:38:23 [WARNING ] (25, 5, 'bot')
2018-11-19 16:38:23 [WARNING ] (25, 5, 'bot')
2018-11-19 16:38:23 [WARNING ] (25, 5, 'bot')
2018-11-19 16:38:23 [WARNING ] (25, 5, 'bot')
2018-11-19 16:38:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:38:23 [DEBUG   ] Finished updating board flags in 1.595 seconds
2018-11-19 16:38:36 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 16:38:56 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 16:39:16 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 16:39:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:39:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:39:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:39:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:39:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:39:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:39:23 [DEBUG   ] Finished updating board flags in 1.413 seconds
2018-11-19 16:39:36 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 16:39:55 [DEBUG   ] Got command RPT from MCS: ref #560297
2018-11-19 16:39:55 [DEBUG   ] summary = ERROR
2018-11-19 16:39:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560297  13 58441 59995745 A  ERRORERROR'
2018-11-19 16:39:55 [DEBUG   ] Got command RPT from MCS: ref #560298
2018-11-19 16:39:55 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:39:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560298  77 58441 59995949 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:39:56 [DEBUG   ] Got command RPT from MCS: ref #560299
2018-11-19 16:39:56 [DEBUG   ] summary = ERROR
2018-11-19 16:39:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560299  13 58441 59996153 A  ERRORERROR'
2018-11-19 16:39:56 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 16:39:56 [DEBUG   ] Got command RPT from MCS: ref #560300
2018-11-19 16:39:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:39:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560300  77 58441 59996345 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:39:56 [DEBUG   ] Got command RPT from MCS: ref #560302
2018-11-19 16:39:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:39:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560302  12 58441 59996549 A  ERRORB˜™š'
2018-11-19 16:40:16 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 16:40:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:40:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:40:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:40:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:40:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:40:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:40:23 [DEBUG   ] Finished updating board flags in 1.556 seconds
2018-11-19 16:40:36 [DEBUG   ] Finished updating ant stats in 0.997 seconds
2018-11-19 16:40:46 [DEBUG   ] Got command RPT from MCS: ref #560311
2018-11-19 16:40:46 [DEBUG   ] summary = ERROR
2018-11-19 16:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560311  13 58441 60046441 A  ERRORERROR'
2018-11-19 16:40:46 [DEBUG   ] Got command RPT from MCS: ref #560312
2018-11-19 16:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560312  77 58441 60046442 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:40:56 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:41:16 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:41:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 16:41:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:41:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:41:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:41:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:41:23 [WARNING ] (13, 1, 'bot')
2018-11-19 16:41:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:41:23 [DEBUG   ] Finished updating board flags in 1.560 seconds
2018-11-19 16:41:36 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:41:56 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:42:16 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 16:42:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 16:42:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:42:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:42:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:42:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:42:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:42:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:42:24 [DEBUG   ] Finished updating board flags in 2.188 seconds
2018-11-19 16:42:36 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:42:56 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:43:16 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:43:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:43:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:43:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:43:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:43:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:43:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:43:24 [DEBUG   ] Finished updating board flags in 1.895 seconds
2018-11-19 16:43:36 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:43:56 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:44:16 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:44:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:44:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:44:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:44:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:44:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:44:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:44:24 [DEBUG   ] Finished updating board flags in 1.805 seconds
2018-11-19 16:44:36 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:44:55 [DEBUG   ] Got command RPT from MCS: ref #560332
2018-11-19 16:44:55 [DEBUG   ] summary = ERROR
2018-11-19 16:44:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560332  13 58441 60295837 A  ERRORERROR'
2018-11-19 16:44:56 [DEBUG   ] Got command RPT from MCS: ref #560333
2018-11-19 16:44:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:44:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560333  77 58441 60296041 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:44:56 [DEBUG   ] Got command RPT from MCS: ref #560334
2018-11-19 16:44:56 [DEBUG   ] summary = ERROR
2018-11-19 16:44:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560334  13 58441 60296233 A  ERRORERROR'
2018-11-19 16:44:56 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:44:56 [DEBUG   ] Got command RPT from MCS: ref #560335
2018-11-19 16:44:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:44:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560335  77 58441 60296437 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:44:56 [DEBUG   ] Got command RPT from MCS: ref #560336
2018-11-19 16:44:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:44:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560336  12 58441 60296641 A  ERRORB˜™š'
2018-11-19 16:45:16 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:45:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:45:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:45:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:45:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:45:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:45:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:45:24 [DEBUG   ] Finished updating board flags in 1.814 seconds
2018-11-19 16:45:36 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:45:46 [DEBUG   ] Got command RPT from MCS: ref #560337
2018-11-19 16:45:46 [DEBUG   ] summary = ERROR
2018-11-19 16:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560337  13 58441 60346437 A  ERRORERROR'
2018-11-19 16:45:46 [DEBUG   ] Got command RPT from MCS: ref #560338
2018-11-19 16:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560338  77 58441 60346438 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:45:56 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:46:16 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 16:46:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:46:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 16:46:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:46:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:46:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:46:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:46:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:46:24 [DEBUG   ] Finished updating board flags in 1.781 seconds
2018-11-19 16:46:36 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:46:56 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:47:16 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:47:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:47:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 16:47:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:47:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:47:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:47:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:47:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:47:24 [DEBUG   ] Finished updating board flags in 1.653 seconds
2018-11-19 16:47:36 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 16:47:56 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:48:16 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:48:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 16:48:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:48:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:48:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:48:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:48:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:48:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:48:24 [DEBUG   ] Finished updating board flags in 1.633 seconds
2018-11-19 16:48:36 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:48:56 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:49:16 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:49:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 16:49:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:49:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:49:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:49:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:49:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:49:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:49:24 [DEBUG   ] Finished updating board flags in 1.552 seconds
2018-11-19 16:49:36 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:49:55 [DEBUG   ] Got command RPT from MCS: ref #560364
2018-11-19 16:49:55 [DEBUG   ] summary = ERROR
2018-11-19 16:49:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560364  13 58441 60595922 A  ERRORERROR'
2018-11-19 16:49:56 [DEBUG   ] Got command RPT from MCS: ref #560365
2018-11-19 16:49:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:49:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560365  77 58441 60596125 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:49:56 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:49:56 [DEBUG   ] Got command RPT from MCS: ref #560366
2018-11-19 16:49:56 [DEBUG   ] summary = ERROR
2018-11-19 16:49:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560366  13 58441 60596329 A  ERRORERROR'
2018-11-19 16:49:56 [DEBUG   ] Got command RPT from MCS: ref #560368
2018-11-19 16:49:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:49:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560368  77 58441 60596521 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:49:56 [DEBUG   ] Got command RPT from MCS: ref #560369
2018-11-19 16:49:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:49:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560369  12 58441 60596725 A  ERRORB™™š'
2018-11-19 16:50:16 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:50:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 16:50:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:50:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 16:50:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:50:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:50:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:50:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:50:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:50:24 [DEBUG   ] Finished updating board flags in 1.540 seconds
2018-11-19 16:50:36 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 16:50:46 [DEBUG   ] Got command RPT from MCS: ref #560374
2018-11-19 16:50:46 [DEBUG   ] summary = ERROR
2018-11-19 16:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560374  13 58441 60646441 A  ERRORERROR'
2018-11-19 16:50:46 [DEBUG   ] Got command RPT from MCS: ref #560375
2018-11-19 16:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560375  77 58441 60646442 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:50:56 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 16:51:16 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 16:51:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:51:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:51:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:51:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:51:24 [WARNING ] (13, 1, 'bot')
2018-11-19 16:51:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:51:24 [DEBUG   ] Finished updating board flags in 1.538 seconds
2018-11-19 16:51:36 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 16:51:56 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 16:52:16 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 16:52:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:52:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:52:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:52:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:52:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:52:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:52:25 [DEBUG   ] Finished updating board flags in 2.156 seconds
2018-11-19 16:52:36 [DEBUG   ] Finished updating ant stats in 0.593 seconds
2018-11-19 16:52:57 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-19 16:53:17 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 16:53:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:53:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:53:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:53:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:53:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:53:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:53:25 [DEBUG   ] Finished updating board flags in 1.944 seconds
2018-11-19 16:53:37 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 16:53:57 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 16:54:17 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 16:54:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:54:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:54:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:54:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:54:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:54:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:54:25 [DEBUG   ] Finished updating board flags in 1.834 seconds
2018-11-19 16:54:37 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 16:54:56 [DEBUG   ] Got command RPT from MCS: ref #560395
2018-11-19 16:54:56 [DEBUG   ] summary = ERROR
2018-11-19 16:54:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560395  13 58441 60896005 A  ERRORERROR'
2018-11-19 16:54:56 [DEBUG   ] Got command RPT from MCS: ref #560396
2018-11-19 16:54:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:54:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560396  77 58441 60896209 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:54:56 [DEBUG   ] Got command RPT from MCS: ref #560397
2018-11-19 16:54:56 [DEBUG   ] summary = ERROR
2018-11-19 16:54:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560397  13 58441 60896413 A  ERRORERROR'
2018-11-19 16:54:56 [DEBUG   ] Got command RPT from MCS: ref #560398
2018-11-19 16:54:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:54:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560398  77 58441 60896605 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:54:56 [DEBUG   ] Got command RPT from MCS: ref #560399
2018-11-19 16:54:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:54:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560399  12 58441 60896809 A  ERRORB˜™š'
2018-11-19 16:54:57 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 16:55:17 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 16:55:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:55:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:55:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:55:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:55:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:55:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:55:25 [DEBUG   ] Finished updating board flags in 1.769 seconds
2018-11-19 16:55:37 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 16:55:46 [DEBUG   ] Got command RPT from MCS: ref #560404
2018-11-19 16:55:46 [DEBUG   ] summary = ERROR
2018-11-19 16:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560404  13 58441 60946445 A  ERRORERROR'
2018-11-19 16:55:46 [DEBUG   ] Got command RPT from MCS: ref #560405
2018-11-19 16:55:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560405  77 58441 60946446 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:55:57 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:56:17 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:56:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:56:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:56:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:56:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:56:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:56:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:56:25 [DEBUG   ] Finished updating board flags in 1.603 seconds
2018-11-19 16:56:37 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:56:57 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:57:17 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:57:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:57:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:57:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:57:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:57:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:57:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:57:25 [DEBUG   ] Finished updating board flags in 1.630 seconds
2018-11-19 16:57:37 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:57:57 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:58:17 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:58:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:58:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:58:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:58:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:58:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:58:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:58:25 [DEBUG   ] Finished updating board flags in 1.672 seconds
2018-11-19 16:58:37 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 16:58:57 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 16:59:17 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 16:59:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 16:59:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:59:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:59:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:59:25 [WARNING ] (13, 1, 'bot')
2018-11-19 16:59:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 16:59:25 [DEBUG   ] Finished updating board flags in 1.501 seconds
2018-11-19 16:59:37 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 16:59:56 [DEBUG   ] Got command RPT from MCS: ref #560427
2018-11-19 16:59:56 [DEBUG   ] summary = ERROR
2018-11-19 16:59:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560427  13 58441 61196097 A  ERRORERROR'
2018-11-19 16:59:56 [DEBUG   ] Got command RPT from MCS: ref #560428
2018-11-19 16:59:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:59:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560428  77 58441 61196301 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:59:56 [DEBUG   ] Got command RPT from MCS: ref #560429
2018-11-19 16:59:56 [DEBUG   ] summary = ERROR
2018-11-19 16:59:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560429  13 58441 61196493 A  ERRORERROR'
2018-11-19 16:59:56 [DEBUG   ] Got command RPT from MCS: ref #560431
2018-11-19 16:59:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 16:59:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560431  77 58441 61196701 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 16:59:56 [DEBUG   ] Got command RPT from MCS: ref #560432
2018-11-19 16:59:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 16:59:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560432  12 58441 61196893 A  ERRORB˜™š'
2018-11-19 16:59:57 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:00:17 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:00:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:00:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:00:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:00:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:00:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:00:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:00:25 [DEBUG   ] Finished updating board flags in 1.494 seconds
2018-11-19 17:00:37 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:00:46 [DEBUG   ] Got command RPT from MCS: ref #560437
2018-11-19 17:00:46 [DEBUG   ] summary = ERROR
2018-11-19 17:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560437  13 58441 61246445 A  ERRORERROR'
2018-11-19 17:00:46 [DEBUG   ] Got command RPT from MCS: ref #560438
2018-11-19 17:00:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560438  77 58441 61246446 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:00:57 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:01:17 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:01:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:01:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:01:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:01:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:01:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:01:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:01:25 [DEBUG   ] Finished updating board flags in 1.497 seconds
2018-11-19 17:01:37 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:01:57 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:02:17 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:02:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:02:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:02:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:02:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:02:25 [WARNING ] (13, 1, 'bot')
2018-11-19 17:02:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:02:25 [DEBUG   ] Finished updating board flags in 1.502 seconds
2018-11-19 17:02:37 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:02:57 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:03:17 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:03:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 17:03:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:03:26 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 17:03:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:03:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:03:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:03:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:03:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:03:26 [DEBUG   ] Finished updating board flags in 2.173 seconds
2018-11-19 17:03:37 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:03:57 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:04:17 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:04:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 17:04:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:04:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 17:04:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:04:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:04:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:04:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:04:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:04:26 [DEBUG   ] Finished updating board flags in 2.029 seconds
2018-11-19 17:04:37 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:04:56 [DEBUG   ] Got command RPT from MCS: ref #560462
2018-11-19 17:04:56 [DEBUG   ] summary = ERROR
2018-11-19 17:04:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560462  13 58441 61496185 A  ERRORERROR'
2018-11-19 17:04:56 [DEBUG   ] Got command RPT from MCS: ref #560463
2018-11-19 17:04:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:04:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560463  77 58441 61496377 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:04:56 [DEBUG   ] Got command RPT from MCS: ref #560464
2018-11-19 17:04:56 [DEBUG   ] summary = ERROR
2018-11-19 17:04:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560464  13 58441 61496582 A  ERRORERROR'
2018-11-19 17:04:56 [DEBUG   ] Got command RPT from MCS: ref #560465
2018-11-19 17:04:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:04:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560465  77 58441 61496786 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:04:56 [DEBUG   ] Got command RPT from MCS: ref #560466
2018-11-19 17:04:56 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:04:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560466  12 58441 61496977 A  ERRORB˜™š'
2018-11-19 17:04:57 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:05:17 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:05:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 17:05:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:05:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:05:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:05:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:05:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:05:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:05:26 [DEBUG   ] Finished updating board flags in 2.016 seconds
2018-11-19 17:05:37 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:05:46 [DEBUG   ] Got command RPT from MCS: ref #560467
2018-11-19 17:05:46 [DEBUG   ] summary = ERROR
2018-11-19 17:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560467  13 58441 61546445 A  ERRORERROR'
2018-11-19 17:05:46 [DEBUG   ] Got command RPT from MCS: ref #560468
2018-11-19 17:05:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560468  77 58441 61546446 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:05:57 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 17:06:17 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 17:06:25 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 17:06:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:06:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:06:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:06:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:06:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:06:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:06:26 [DEBUG   ] Finished updating board flags in 1.889 seconds
2018-11-19 17:06:37 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:06:57 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:07:17 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:07:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:07:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:07:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:07:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:07:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:07:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:07:26 [DEBUG   ] Finished updating board flags in 1.829 seconds
2018-11-19 17:07:37 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:07:57 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:08:17 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:08:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:08:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:08:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:08:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:08:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:08:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:08:26 [DEBUG   ] Finished updating board flags in 1.785 seconds
2018-11-19 17:08:37 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:08:57 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:09:17 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:09:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:09:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:09:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:09:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:09:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:09:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:09:26 [DEBUG   ] Finished updating board flags in 1.633 seconds
2018-11-19 17:09:37 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:09:56 [DEBUG   ] Got command RPT from MCS: ref #560490
2018-11-19 17:09:56 [DEBUG   ] summary = ERROR
2018-11-19 17:09:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560490  13 58441 61796265 A  ERRORERROR'
2018-11-19 17:09:56 [DEBUG   ] Got command RPT from MCS: ref #560491
2018-11-19 17:09:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:09:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560491  77 58441 61796469 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:09:56 [DEBUG   ] Got command RPT from MCS: ref #560492
2018-11-19 17:09:56 [DEBUG   ] summary = ERROR
2018-11-19 17:09:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560492  13 58441 61796661 A  ERRORERROR'
2018-11-19 17:09:56 [DEBUG   ] Got command RPT from MCS: ref #560494
2018-11-19 17:09:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:09:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560494  77 58441 61796865 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:09:57 [DEBUG   ] Got command RPT from MCS: ref #560496
2018-11-19 17:09:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:09:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560496  12 58441 61797069 A  ERRORB™™š'
2018-11-19 17:09:57 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:10:17 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:10:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 17:10:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:10:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:10:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:10:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:10:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:10:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:10:26 [DEBUG   ] Finished updating board flags in 1.655 seconds
2018-11-19 17:10:37 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:10:46 [DEBUG   ] Got command RPT from MCS: ref #560504
2018-11-19 17:10:46 [DEBUG   ] summary = ERROR
2018-11-19 17:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560504  13 58441 61846445 A  ERRORERROR'
2018-11-19 17:10:46 [DEBUG   ] Got command RPT from MCS: ref #560505
2018-11-19 17:10:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560505  77 58441 61846446 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:10:57 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:11:17 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:11:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:11:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:11:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:11:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:11:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:11:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:11:26 [DEBUG   ] Finished updating board flags in 1.625 seconds
2018-11-19 17:11:37 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:11:57 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:12:17 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:12:25 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 17:12:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:12:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:12:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:12:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:12:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:12:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:12:26 [DEBUG   ] Finished updating board flags in 1.499 seconds
2018-11-19 17:12:37 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 17:12:57 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:13:17 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:13:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:13:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:13:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:13:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:13:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:13:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:13:26 [DEBUG   ] Finished updating board flags in 1.484 seconds
2018-11-19 17:13:37 [DEBUG   ] Finished updating ant stats in 0.597 seconds
2018-11-19 17:13:58 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-19 17:14:18 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 17:14:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:14:26 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 17:14:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:14:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:14:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:14:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:14:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:14:26 [DEBUG   ] Finished updating board flags in 1.503 seconds
2018-11-19 17:14:38 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 17:14:56 [DEBUG   ] Got command RPT from MCS: ref #560525
2018-11-19 17:14:56 [DEBUG   ] summary = ERROR
2018-11-19 17:14:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560525  13 58441 62096349 A  ERRORERROR'
2018-11-19 17:14:56 [DEBUG   ] Got command RPT from MCS: ref #560526
2018-11-19 17:14:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:14:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560526  77 58441 62096553 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:14:56 [DEBUG   ] Got command RPT from MCS: ref #560527
2018-11-19 17:14:56 [DEBUG   ] summary = ERROR
2018-11-19 17:14:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560527  13 58441 62096757 A  ERRORERROR'
2018-11-19 17:14:56 [DEBUG   ] Got command RPT from MCS: ref #560528
2018-11-19 17:14:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:14:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560528  77 58441 62096949 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:14:57 [DEBUG   ] Got command RPT from MCS: ref #560529
2018-11-19 17:14:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:14:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560529  12 58441 62097154 A  ERRORB˜™š'
2018-11-19 17:14:58 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 17:15:18 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 17:15:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:15:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:15:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:15:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:15:26 [WARNING ] (13, 1, 'bot')
2018-11-19 17:15:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:15:26 [DEBUG   ] Finished updating board flags in 1.559 seconds
2018-11-19 17:15:38 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 17:15:46 [DEBUG   ] Got command RPT from MCS: ref #560530
2018-11-19 17:15:46 [DEBUG   ] summary = ERROR
2018-11-19 17:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560530  13 58441 62146449 A  ERRORERROR'
2018-11-19 17:15:46 [DEBUG   ] Got command RPT from MCS: ref #560531
2018-11-19 17:15:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560531  77 58441 62146450 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:15:58 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:16:18 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:16:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:16:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:16:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:16:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:16:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:16:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:16:27 [DEBUG   ] Finished updating board flags in 2.115 seconds
2018-11-19 17:16:38 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:16:58 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:17:18 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:17:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:17:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:17:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:17:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:17:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:17:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:17:27 [DEBUG   ] Finished updating board flags in 1.942 seconds
2018-11-19 17:17:38 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 17:17:58 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:18:18 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:18:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:18:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:18:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:18:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:18:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:18:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:18:27 [DEBUG   ] Finished updating board flags in 1.830 seconds
2018-11-19 17:18:38 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:18:58 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:19:18 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:19:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:19:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:19:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:19:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:19:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:19:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:19:27 [DEBUG   ] Finished updating board flags in 1.866 seconds
2018-11-19 17:19:38 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:19:56 [DEBUG   ] Got command RPT from MCS: ref #560557
2018-11-19 17:19:56 [DEBUG   ] summary = ERROR
2018-11-19 17:19:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560557  13 58441 62396441 A  ERRORERROR'
2018-11-19 17:19:56 [DEBUG   ] Got command RPT from MCS: ref #560558
2018-11-19 17:19:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:19:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560558  77 58441 62396633 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:19:56 [DEBUG   ] Got command RPT from MCS: ref #560559
2018-11-19 17:19:56 [DEBUG   ] summary = ERROR
2018-11-19 17:19:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560559  13 58441 62396837 A  ERRORERROR'
2018-11-19 17:19:57 [DEBUG   ] Got command RPT from MCS: ref #560561
2018-11-19 17:19:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:19:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560561  77 58441 62397041 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:19:57 [DEBUG   ] Got command RPT from MCS: ref #560562
2018-11-19 17:19:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:19:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560562  12 58441 62397233 A  ERRORB˜™š'
2018-11-19 17:19:58 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:20:18 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 17:20:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:20:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 17:20:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:20:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:20:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:20:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:20:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:20:27 [DEBUG   ] Finished updating board flags in 1.720 seconds
2018-11-19 17:20:38 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:20:46 [DEBUG   ] Got command RPT from MCS: ref #560567
2018-11-19 17:20:46 [DEBUG   ] summary = ERROR
2018-11-19 17:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560567  13 58441 62446453 A  ERRORERROR'
2018-11-19 17:20:46 [DEBUG   ] Got command RPT from MCS: ref #560568
2018-11-19 17:20:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560568  77 58441 62446454 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:20:58 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:21:18 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:21:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:21:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:21:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:21:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:21:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:21:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:21:27 [DEBUG   ] Finished updating board flags in 1.603 seconds
2018-11-19 17:21:38 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 17:21:58 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:22:18 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:22:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:22:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:22:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:22:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:22:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:22:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:22:27 [DEBUG   ] Finished updating board flags in 1.590 seconds
2018-11-19 17:22:38 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:22:58 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:23:18 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:23:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:23:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:23:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:23:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:23:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:23:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:23:27 [DEBUG   ] Finished updating board flags in 1.473 seconds
2018-11-19 17:23:38 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:23:58 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:24:18 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:24:26 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 17:24:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:24:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 17:24:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 17:24:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:24:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:24:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:24:27 [WARNING ] (13, 1, 'bot')
2018-11-19 17:24:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:24:27 [DEBUG   ] Finished updating board flags in 1.559 seconds
2018-11-19 17:24:38 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:24:56 [DEBUG   ] Got command RPT from MCS: ref #560588
2018-11-19 17:24:56 [DEBUG   ] summary = ERROR
2018-11-19 17:24:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560588  13 58441 62696517 A  ERRORERROR'
2018-11-19 17:24:56 [DEBUG   ] Got command RPT from MCS: ref #560589
2018-11-19 17:24:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:24:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560589  77 58441 62696721 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:24:56 [DEBUG   ] Got command RPT from MCS: ref #560590
2018-11-19 17:24:56 [DEBUG   ] summary = ERROR
2018-11-19 17:24:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560590  13 58441 62696925 A  ERRORERROR'
2018-11-19 17:24:57 [DEBUG   ] Got command RPT from MCS: ref #560591
2018-11-19 17:24:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:24:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560591  77 58441 62697129 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:24:57 [DEBUG   ] Got command RPT from MCS: ref #560592
2018-11-19 17:24:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:24:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560592  12 58441 62697321 A  ERRORB˜™š'
2018-11-19 17:24:58 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:25:18 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:25:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:25:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 17:25:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:25:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:25:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:25:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:25:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:25:28 [DEBUG   ] Finished updating board flags in 2.302 seconds
2018-11-19 17:25:38 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:25:46 [DEBUG   ] Got command RPT from MCS: ref #560597
2018-11-19 17:25:46 [DEBUG   ] summary = ERROR
2018-11-19 17:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560597  13 58441 62746449 A  ERRORERROR'
2018-11-19 17:25:46 [DEBUG   ] Got command RPT from MCS: ref #560598
2018-11-19 17:25:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560598  77 58441 62746450 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:25:58 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:26:18 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:26:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:26:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:26:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:26:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:26:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:26:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:26:28 [DEBUG   ] Finished updating board flags in 2.075 seconds
2018-11-19 17:26:38 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:26:58 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:27:18 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:27:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:27:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 17:27:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:27:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:27:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:27:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:27:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:27:28 [DEBUG   ] Finished updating board flags in 2.021 seconds
2018-11-19 17:27:38 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:27:58 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:28:18 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 17:28:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 17:28:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:28:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:28:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:28:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:28:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:28:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:28:28 [DEBUG   ] Finished updating board flags in 1.992 seconds
2018-11-19 17:28:38 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:28:58 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 17:29:18 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:29:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:29:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:29:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:29:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:29:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:29:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:29:28 [DEBUG   ] Finished updating board flags in 1.827 seconds
2018-11-19 17:29:38 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:29:56 [DEBUG   ] Got command RPT from MCS: ref #560620
2018-11-19 17:29:56 [DEBUG   ] summary = ERROR
2018-11-19 17:29:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560620  13 58441 62996605 A  ERRORERROR'
2018-11-19 17:29:56 [DEBUG   ] Got command RPT from MCS: ref #560621
2018-11-19 17:29:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:29:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560621  77 58441 62996809 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:29:57 [DEBUG   ] Got command RPT from MCS: ref #560622
2018-11-19 17:29:57 [DEBUG   ] summary = ERROR
2018-11-19 17:29:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560622  13 58441 62997013 A  ERRORERROR'
2018-11-19 17:29:57 [DEBUG   ] Got command RPT from MCS: ref #560624
2018-11-19 17:29:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:29:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560624  77 58441 62997205 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:29:57 [DEBUG   ] Got command RPT from MCS: ref #560626
2018-11-19 17:29:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:29:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560626  12 58441 62997409 A  ERRORB™™š'
2018-11-19 17:29:58 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:30:18 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 17:30:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:30:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 17:30:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:30:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:30:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:30:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:30:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:30:28 [DEBUG   ] Finished updating board flags in 1.880 seconds
2018-11-19 17:30:38 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 17:30:46 [DEBUG   ] Got command RPT from MCS: ref #560630
2018-11-19 17:30:46 [DEBUG   ] summary = ERROR
2018-11-19 17:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560630  13 58441 63046449 A  ERRORERROR'
2018-11-19 17:30:46 [DEBUG   ] Got command RPT from MCS: ref #560631
2018-11-19 17:30:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560631  77 58441 63046450 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:30:58 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 17:31:19 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-19 17:31:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:31:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:31:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:31:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:31:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:31:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:31:28 [DEBUG   ] Finished updating board flags in 1.705 seconds
2018-11-19 17:31:39 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 17:31:59 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 17:32:19 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 17:32:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 17:32:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:32:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:32:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:32:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:32:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:32:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:32:28 [DEBUG   ] Finished updating board flags in 1.608 seconds
2018-11-19 17:32:39 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-19 17:32:59 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:33:19 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:33:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:33:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:33:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:33:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:33:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:33:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:33:28 [DEBUG   ] Finished updating board flags in 1.587 seconds
2018-11-19 17:33:39 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:33:59 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:34:19 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:34:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:34:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:34:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:34:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:34:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:34:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:34:28 [DEBUG   ] Finished updating board flags in 1.444 seconds
2018-11-19 17:34:39 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:34:56 [DEBUG   ] Got command RPT from MCS: ref #560655
2018-11-19 17:34:56 [DEBUG   ] summary = ERROR
2018-11-19 17:34:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560655  13 58441 63296689 A  ERRORERROR'
2018-11-19 17:34:56 [DEBUG   ] Got command RPT from MCS: ref #560656
2018-11-19 17:34:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:34:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560656  77 58441 63296893 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:34:57 [DEBUG   ] Got command RPT from MCS: ref #560657
2018-11-19 17:34:57 [DEBUG   ] summary = ERROR
2018-11-19 17:34:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560657  13 58441 63297097 A  ERRORERROR'
2018-11-19 17:34:57 [DEBUG   ] Got command RPT from MCS: ref #560658
2018-11-19 17:34:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:34:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560658  77 58441 63297289 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:34:57 [DEBUG   ] Got command RPT from MCS: ref #560659
2018-11-19 17:34:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:34:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560659  12 58441 63297493 A  ERRORB˜™š'
2018-11-19 17:34:59 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:35:19 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:35:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:35:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:35:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:35:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:35:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:35:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:35:28 [DEBUG   ] Finished updating board flags in 1.500 seconds
2018-11-19 17:35:39 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:35:46 [DEBUG   ] Got command RPT from MCS: ref #560660
2018-11-19 17:35:46 [DEBUG   ] summary = ERROR
2018-11-19 17:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560660  13 58441 63346449 A  ERRORERROR'
2018-11-19 17:35:46 [DEBUG   ] Got command RPT from MCS: ref #560661
2018-11-19 17:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560661  77 58441 63346450 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:35:59 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:36:19 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:36:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:36:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:36:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:36:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:36:28 [WARNING ] (13, 1, 'bot')
2018-11-19 17:36:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:36:28 [DEBUG   ] Finished updating board flags in 1.502 seconds
2018-11-19 17:36:39 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:36:59 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 17:37:19 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:37:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:37:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:37:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:37:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:37:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:37:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:37:29 [DEBUG   ] Finished updating board flags in 2.248 seconds
2018-11-19 17:37:39 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:37:59 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:38:19 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:38:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:38:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:38:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:38:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:38:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:38:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:38:29 [DEBUG   ] Finished updating board flags in 2.246 seconds
2018-11-19 17:38:39 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:38:59 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:39:19 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:39:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 17:39:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:39:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:39:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:39:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:39:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:39:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:39:29 [DEBUG   ] Finished updating board flags in 2.086 seconds
2018-11-19 17:39:39 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:39:56 [DEBUG   ] Got command RPT from MCS: ref #560683
2018-11-19 17:39:56 [DEBUG   ] summary = ERROR
2018-11-19 17:39:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560683  13 58441 63596777 A  ERRORERROR'
2018-11-19 17:39:56 [DEBUG   ] Got command RPT from MCS: ref #560684
2018-11-19 17:39:56 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:39:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560684  77 58441 63596981 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:39:57 [DEBUG   ] Got command RPT from MCS: ref #560685
2018-11-19 17:39:57 [DEBUG   ] summary = ERROR
2018-11-19 17:39:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560685  13 58441 63597173 A  ERRORERROR'
2018-11-19 17:39:57 [DEBUG   ] Got command RPT from MCS: ref #560687
2018-11-19 17:39:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:39:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560687  77 58441 63597377 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:39:57 [DEBUG   ] Got command RPT from MCS: ref #560689
2018-11-19 17:39:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:39:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560689  12 58441 63597581 A  ERRORB˜™š'
2018-11-19 17:39:59 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:40:19 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:40:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:40:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:40:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:40:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:40:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:40:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:40:29 [DEBUG   ] Finished updating board flags in 2.027 seconds
2018-11-19 17:40:39 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:40:46 [DEBUG   ] Got command RPT from MCS: ref #560697
2018-11-19 17:40:46 [DEBUG   ] summary = ERROR
2018-11-19 17:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560697  13 58441 63646461 A  ERRORERROR'
2018-11-19 17:40:46 [DEBUG   ] Got command RPT from MCS: ref #560698
2018-11-19 17:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560698  77 58441 63646462 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:40:59 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 17:41:19 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 17:41:28 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 17:41:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:41:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:41:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:41:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:41:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:41:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:41:29 [DEBUG   ] Finished updating board flags in 1.985 seconds
2018-11-19 17:41:39 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:41:59 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:42:19 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 17:42:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:42:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 17:42:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:42:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:42:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:42:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:42:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:42:29 [DEBUG   ] Finished updating board flags in 1.805 seconds
2018-11-19 17:42:39 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:42:59 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:43:19 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:43:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:43:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:43:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:43:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:43:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:43:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:43:29 [DEBUG   ] Finished updating board flags in 1.782 seconds
2018-11-19 17:43:39 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:43:59 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:44:19 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:44:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:44:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:44:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:44:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:44:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:44:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:44:29 [DEBUG   ] Finished updating board flags in 1.697 seconds
2018-11-19 17:44:39 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:44:56 [DEBUG   ] Got command RPT from MCS: ref #560718
2018-11-19 17:44:56 [DEBUG   ] summary = ERROR
2018-11-19 17:44:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560718  13 58441 63896857 A  ERRORERROR'
2018-11-19 17:44:57 [DEBUG   ] Got command RPT from MCS: ref #560719
2018-11-19 17:44:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:44:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560719  77 58441 63897061 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:44:57 [DEBUG   ] Got command RPT from MCS: ref #560720
2018-11-19 17:44:57 [DEBUG   ] summary = ERROR
2018-11-19 17:44:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560720  13 58441 63897265 A  ERRORERROR'
2018-11-19 17:44:57 [DEBUG   ] Got command RPT from MCS: ref #560721
2018-11-19 17:44:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:44:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560721  77 58441 63897469 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:44:57 [DEBUG   ] Got command RPT from MCS: ref #560722
2018-11-19 17:44:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:44:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560722  12 58441 63897661 A  ERRORB—™š'
2018-11-19 17:44:59 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 17:45:19 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 17:45:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:45:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:45:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:45:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:45:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:45:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:45:29 [DEBUG   ] Finished updating board flags in 1.508 seconds
2018-11-19 17:45:39 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-19 17:45:46 [DEBUG   ] Got command RPT from MCS: ref #560723
2018-11-19 17:45:46 [DEBUG   ] summary = ERROR
2018-11-19 17:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560723  13 58441 63946461 A  ERRORERROR'
2018-11-19 17:45:46 [DEBUG   ] Got command RPT from MCS: ref #560724
2018-11-19 17:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560724  77 58441 63946462 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:45:59 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 17:46:19 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 17:46:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:46:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:46:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:46:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:46:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:46:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:46:29 [DEBUG   ] Finished updating board flags in 1.485 seconds
2018-11-19 17:46:39 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 17:47:00 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-19 17:47:20 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-19 17:47:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:47:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:47:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:47:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:47:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:47:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:47:29 [DEBUG   ] Finished updating board flags in 1.478 seconds
2018-11-19 17:47:40 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 17:48:00 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:48:20 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:48:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 17:48:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:48:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:48:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:48:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:48:29 [WARNING ] (13, 1, 'bot')
2018-11-19 17:48:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:48:29 [DEBUG   ] Finished updating board flags in 1.480 seconds
2018-11-19 17:48:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:49:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:49:20 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:49:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 17:49:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:49:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 17:49:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:49:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:49:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:49:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:49:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:49:30 [DEBUG   ] Finished updating board flags in 2.199 seconds
2018-11-19 17:49:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:49:56 [DEBUG   ] Got command RPT from MCS: ref #560750
2018-11-19 17:49:56 [DEBUG   ] summary = ERROR
2018-11-19 17:49:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560750  13 58441 64196945 A  ERRORERROR'
2018-11-19 17:49:57 [DEBUG   ] Got command RPT from MCS: ref #560751
2018-11-19 17:49:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:49:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560751  77 58441 64197149 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:49:57 [DEBUG   ] Got command RPT from MCS: ref #560752
2018-11-19 17:49:57 [DEBUG   ] summary = ERROR
2018-11-19 17:49:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560752  13 58441 64197353 A  ERRORERROR'
2018-11-19 17:49:57 [DEBUG   ] Got command RPT from MCS: ref #560754
2018-11-19 17:49:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:49:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560754  77 58441 64197545 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:49:57 [DEBUG   ] Got command RPT from MCS: ref #560756
2018-11-19 17:49:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:49:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560756  12 58441 64197749 A  ERRORB™™š'
2018-11-19 17:50:00 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 17:50:20 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:50:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:50:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 17:50:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:50:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:50:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:50:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:50:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:50:30 [DEBUG   ] Finished updating board flags in 2.093 seconds
2018-11-19 17:50:40 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:50:46 [DEBUG   ] Got command RPT from MCS: ref #560760
2018-11-19 17:50:46 [DEBUG   ] summary = ERROR
2018-11-19 17:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560760  13 58441 64246453 A  ERRORERROR'
2018-11-19 17:50:46 [DEBUG   ] Got command RPT from MCS: ref #560761
2018-11-19 17:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560761  77 58441 64246454 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:51:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:51:20 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 17:51:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:51:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 17:51:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 17:51:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:51:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:51:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:51:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:51:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:51:30 [DEBUG   ] Finished updating board flags in 1.877 seconds
2018-11-19 17:51:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:52:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:52:20 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:52:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 17:52:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:52:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:52:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:52:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:52:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:52:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:52:30 [DEBUG   ] Finished updating board flags in 1.813 seconds
2018-11-19 17:52:40 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 17:53:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:53:20 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 17:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:53:30 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 17:53:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:53:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:53:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:53:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:53:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:53:30 [DEBUG   ] Finished updating board flags in 1.808 seconds
2018-11-19 17:53:40 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:54:00 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 17:54:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 17:54:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:54:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 17:54:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:54:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:54:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:54:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:54:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:54:30 [DEBUG   ] Finished updating board flags in 1.785 seconds
2018-11-19 17:54:40 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 17:54:57 [DEBUG   ] Got command RPT from MCS: ref #560781
2018-11-19 17:54:57 [DEBUG   ] summary = ERROR
2018-11-19 17:54:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560781  13 58441 64497037 A  ERRORERROR'
2018-11-19 17:54:57 [DEBUG   ] Got command RPT from MCS: ref #560782
2018-11-19 17:54:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:54:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560782  77 58441 64497229 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:54:57 [DEBUG   ] Got command RPT from MCS: ref #560783
2018-11-19 17:54:57 [DEBUG   ] summary = ERROR
2018-11-19 17:54:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560783  13 58441 64497433 A  ERRORERROR'
2018-11-19 17:54:57 [DEBUG   ] Got command RPT from MCS: ref #560784
2018-11-19 17:54:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:54:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560784  77 58441 64497637 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:54:57 [DEBUG   ] Got command RPT from MCS: ref #560785
2018-11-19 17:54:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:54:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560785  12 58441 64497841 A  ERRORB™™š'
2018-11-19 17:55:00 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 17:55:20 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 17:55:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:55:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:55:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:55:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:55:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:55:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:55:30 [DEBUG   ] Finished updating board flags in 1.666 seconds
2018-11-19 17:55:40 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:55:46 [DEBUG   ] Got command RPT from MCS: ref #560790
2018-11-19 17:55:46 [DEBUG   ] summary = ERROR
2018-11-19 17:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560790  13 58441 64546457 A  ERRORERROR'
2018-11-19 17:55:46 [DEBUG   ] Got command RPT from MCS: ref #560791
2018-11-19 17:55:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560791  77 58441 64546458 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:56:00 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:56:20 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:56:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:56:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:56:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:56:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:56:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:56:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:56:30 [DEBUG   ] Finished updating board flags in 1.610 seconds
2018-11-19 17:56:40 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:57:00 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:57:20 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:57:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:57:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:57:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:57:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:57:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:57:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:57:30 [DEBUG   ] Finished updating board flags in 1.636 seconds
2018-11-19 17:57:40 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:58:00 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:58:20 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:58:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:58:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:58:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:58:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:58:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:58:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:58:30 [DEBUG   ] Finished updating board flags in 1.515 seconds
2018-11-19 17:58:40 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 17:59:00 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 17:59:20 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:59:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 17:59:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:59:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:59:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:59:30 [WARNING ] (13, 1, 'bot')
2018-11-19 17:59:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 17:59:30 [DEBUG   ] Finished updating board flags in 1.510 seconds
2018-11-19 17:59:40 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 17:59:57 [DEBUG   ] Got command RPT from MCS: ref #560813
2018-11-19 17:59:57 [DEBUG   ] summary = ERROR
2018-11-19 17:59:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560813  13 58441 64797121 A  ERRORERROR'
2018-11-19 17:59:57 [DEBUG   ] Got command RPT from MCS: ref #560814
2018-11-19 17:59:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:59:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560814  77 58441 64797313 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:59:57 [DEBUG   ] Got command RPT from MCS: ref #560815
2018-11-19 17:59:57 [DEBUG   ] summary = ERROR
2018-11-19 17:59:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560815  13 58441 64797517 A  ERRORERROR'
2018-11-19 17:59:57 [DEBUG   ] Got command RPT from MCS: ref #560817
2018-11-19 17:59:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 17:59:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560817  77 58441 64797721 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 17:59:57 [DEBUG   ] Got command RPT from MCS: ref #560818
2018-11-19 17:59:57 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 17:59:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560818  12 58441 64797925 A  ERRORB˜™š'
2018-11-19 18:00:00 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 18:00:20 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:00:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:00:30 [WARNING ] (13, 1, 'bot')
2018-11-19 18:00:30 [WARNING ] (13, 1, 'bot')
2018-11-19 18:00:30 [WARNING ] (13, 1, 'bot')
2018-11-19 18:00:30 [WARNING ] (13, 1, 'bot')
2018-11-19 18:00:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:00:30 [DEBUG   ] Finished updating board flags in 1.521 seconds
2018-11-19 18:00:40 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-19 18:00:46 [DEBUG   ] Got command RPT from MCS: ref #560823
2018-11-19 18:00:46 [DEBUG   ] summary = ERROR
2018-11-19 18:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560823  13 58441 64846465 A  ERRORERROR'
2018-11-19 18:00:46 [DEBUG   ] Got command RPT from MCS: ref #560824
2018-11-19 18:00:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560824  77 58441 64846466 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:01:00 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 18:01:20 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:01:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:01:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:01:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:01:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:01:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:01:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:01:31 [DEBUG   ] Finished updating board flags in 2.196 seconds
2018-11-19 18:01:40 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 18:02:00 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 18:02:20 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:02:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:02:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:02:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:02:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:02:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:02:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:02:31 [DEBUG   ] Finished updating board flags in 1.959 seconds
2018-11-19 18:02:40 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:03:00 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:03:20 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 18:03:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:03:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 18:03:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:03:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:03:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:03:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:03:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:03:31 [DEBUG   ] Finished updating board flags in 1.836 seconds
2018-11-19 18:03:40 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 18:04:00 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 18:04:20 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:04:30 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 18:04:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:04:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 18:04:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:04:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:04:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:04:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:04:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:04:31 [DEBUG   ] Finished updating board flags in 1.870 seconds
2018-11-19 18:04:40 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 18:04:57 [DEBUG   ] Got command RPT from MCS: ref #560848
2018-11-19 18:04:57 [DEBUG   ] summary = ERROR
2018-11-19 18:04:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560848  13 58441 65097205 A  ERRORERROR'
2018-11-19 18:04:57 [DEBUG   ] Got command RPT from MCS: ref #560849
2018-11-19 18:04:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:04:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560849  77 58441 65097409 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:04:57 [DEBUG   ] Got command RPT from MCS: ref #560850
2018-11-19 18:04:57 [DEBUG   ] summary = ERROR
2018-11-19 18:04:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560850  13 58441 65097601 A  ERRORERROR'
2018-11-19 18:04:57 [DEBUG   ] Got command RPT from MCS: ref #560851
2018-11-19 18:04:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:04:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560851  77 58441 65097805 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:04:58 [DEBUG   ] Got command RPT from MCS: ref #560852
2018-11-19 18:04:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:04:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560852  12 58441 65098009 A  ERRORB˜™š'
2018-11-19 18:05:01 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-19 18:05:21 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-19 18:05:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:05:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 18:05:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:05:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:05:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:05:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:05:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:05:31 [DEBUG   ] Finished updating board flags in 1.808 seconds
2018-11-19 18:05:41 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:05:46 [DEBUG   ] Got command RPT from MCS: ref #560853
2018-11-19 18:05:46 [DEBUG   ] summary = ERROR
2018-11-19 18:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560853  13 58441 65146465 A  ERRORERROR'
2018-11-19 18:05:46 [DEBUG   ] Got command RPT from MCS: ref #560854
2018-11-19 18:05:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560854  77 58441 65146466 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:06:01 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:06:21 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 18:06:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:06:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:06:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:06:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:06:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:06:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:06:31 [DEBUG   ] Finished updating board flags in 1.696 seconds
2018-11-19 18:06:41 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:07:01 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:07:21 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:07:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:07:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:07:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:07:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:07:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:07:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:07:31 [DEBUG   ] Finished updating board flags in 1.616 seconds
2018-11-19 18:07:41 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 18:08:01 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:08:21 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 18:08:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:08:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:08:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:08:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:08:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:08:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:08:31 [DEBUG   ] Finished updating board flags in 1.617 seconds
2018-11-19 18:08:41 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:09:01 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 18:09:21 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:09:30 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 18:09:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:09:31 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 18:09:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:09:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:09:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:09:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:09:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:09:31 [DEBUG   ] Finished updating board flags in 1.613 seconds
2018-11-19 18:09:41 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:09:57 [DEBUG   ] Got command RPT from MCS: ref #560876
2018-11-19 18:09:57 [DEBUG   ] summary = ERROR
2018-11-19 18:09:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560876  13 58441 65397297 A  ERRORERROR'
2018-11-19 18:09:57 [DEBUG   ] Got command RPT from MCS: ref #560877
2018-11-19 18:09:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:09:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560877  77 58441 65397489 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:09:57 [DEBUG   ] Got command RPT from MCS: ref #560878
2018-11-19 18:09:57 [DEBUG   ] summary = ERROR
2018-11-19 18:09:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560878  13 58441 65397693 A  ERRORERROR'
2018-11-19 18:09:57 [DEBUG   ] Got command RPT from MCS: ref #560880
2018-11-19 18:09:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:09:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560880  77 58441 65397897 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:09:58 [DEBUG   ] Got command RPT from MCS: ref #560881
2018-11-19 18:09:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:09:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560881  12 58441 65398089 A  ERRORB—™š'
2018-11-19 18:10:01 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:10:21 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:10:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:10:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:10:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:10:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:10:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:10:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:10:31 [DEBUG   ] Finished updating board flags in 1.659 seconds
2018-11-19 18:10:41 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 18:10:46 [DEBUG   ] Got command RPT from MCS: ref #560890
2018-11-19 18:10:46 [DEBUG   ] summary = ERROR
2018-11-19 18:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560890  13 58441 65446465 A  ERRORERROR'
2018-11-19 18:10:46 [DEBUG   ] Got command RPT from MCS: ref #560891
2018-11-19 18:10:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560891  77 58441 65446466 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:11:01 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 18:11:12 [DEBUG   ] Got command TBW from MCS: ref #560908
2018-11-19 18:11:12 [DEBUG   ] tbw = exited with status True, code 0x00
2018-11-19 18:11:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_TBW   560908   8 58441 65472018 A  ERROR'
2018-11-19 18:11:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 pkill tbn
2018-11-19 18:11:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa27 command /usr/local/bin/lwa27 pkill tbn
2018-11-19 18:11:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa24 command /usr/local/bin/lwa24 pkill tbn
2018-11-19 18:11:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa23 command /usr/local/bin/lwa23 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa18 command /usr/local/bin/lwa18 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa10 command /usr/local/bin/lwa10 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa32 command /usr/local/bin/lwa32 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa11 command /usr/local/bin/lwa11 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa30 command /usr/local/bin/lwa30 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa12 command /usr/local/bin/lwa12 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa8 command /usr/local/bin/lwa8 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa13 command /usr/local/bin/lwa13 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa26 command /usr/local/bin/lwa26 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa14 command /usr/local/bin/lwa14 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa22 command /usr/local/bin/lwa22 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa20 command /usr/local/bin/lwa20 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa1 command /usr/local/bin/lwa1 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa5 command /usr/local/bin/lwa5 pkill tbn
2018-11-19 18:11:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa25 command /usr/local/bin/lwa25 pkill tbn
2018-11-19 18:11:15 [DEBUG   ] sendHostnameCommand - Sent hostname lwa35 command /usr/local/bin/lwa35 pkill tbn
2018-11-19 18:11:15 [DEBUG   ] sendHostnameCommand - Sent hostname lwa19 command /usr/local/bin/lwa19 pkill tbn
2018-11-19 18:11:15 [DEBUG   ] sendHostnameCommand - Sent hostname lwa21 command /usr/local/bin/lwa21 pkill tbn
2018-11-19 18:11:15 [DEBUG   ] sendHostnameCommand - Sent hostname lwa6 command /usr/local/bin/lwa6 pkill tbn
2018-11-19 18:11:15 [DEBUG   ] sendHostnameCommand - Sent hostname lwa7 command /usr/local/bin/lwa7 pkill tbn
2018-11-19 18:11:15 [DEBUG   ] sendHostnameCommand - Sent hostname lwa33 command /usr/local/bin/lwa33 pkill tbn
2018-11-19 18:11:15 [DEBUG   ] sendHostnameCommand - Sent hostname lwa15 command /usr/local/bin/lwa15 pkill tbn
2018-11-19 18:11:15 [DEBUG   ] sendHostnameCommand - Sent hostname lwa28 command /usr/local/bin/lwa28 pkill tbn
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa27', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '1', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa24', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '2', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa23', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '3', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa18', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '4', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa10', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '5', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa32', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '6', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa11', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '7', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa30', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '8', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa12', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '9', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa8', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '10', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa13', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '11', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa26', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '12', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa14', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '13', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa22', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '14', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa20', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '15', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa1', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '16', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa5', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '17', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa25', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '18', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa35', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '19', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa19', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '20', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa21', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '21', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa6', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '22', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa7', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '23', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa33', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '24', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa15', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '25', '12000000']
2018-11-19 18:11:15 [DEBUG   ] spawn_process: at 2018-11-19 18:11:15 running ['/usr/local/bin/lwa28', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '26', '12000000']
2018-11-19 18:11:15 [INFO    ] Finished spawning TBW
2018-11-19 18:11:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:11:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:11:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:11:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:11:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:11:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:11:31 [DEBUG   ] Finished updating board flags in 1.337 seconds
2018-11-19 18:12:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:12:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:12:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:12:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:12:31 [WARNING ] (13, 1, 'bot')
2018-11-19 18:12:31 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:12:31 [DEBUG   ] Finished updating board flags in 1.324 seconds
2018-11-19 18:13:03 [INFO    ] __waitForTBW - TBW finished in 108.282 s
2018-11-19 18:13:04 [DEBUG   ] Finished updating ant stats in 0.451 seconds
2018-11-19 18:13:04 [INFO    ] Started antennas statistics background thread
2018-11-19 18:13:04 [INFO    ] Resuming TBN at 38.100 MHz after TBW acquisition
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa27', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '1', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa24', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '2', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa23', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '3', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa18', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '4', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa10', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '5', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa32', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '6', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa11', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '7', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa30', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '8', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa12', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '9', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa8', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '10', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa13', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '11', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa26', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '12', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa14', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '13', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa22', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '14', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa20', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '15', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa1', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '16', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa5', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '17', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa25', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '18', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa35', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '19', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa19', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '20', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa21', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '21', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa6', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '22', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa7', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '23', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa33', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '24', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa15', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '25', '1542651186']
2018-11-19 18:13:04 [DEBUG   ] spawn_process: at 2018-11-19 18:13:04 running ['/usr/local/bin/lwa28', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '26', '1542651186']
2018-11-19 18:13:04 [INFO    ] Finished spawning TBN in 0.050 s
2018-11-19 18:13:24 [DEBUG   ] Finished updating ant stats in 0.470 seconds
2018-11-19 18:13:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:13:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 18:13:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:13:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:13:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:13:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:13:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:13:32 [DEBUG   ] Finished updating board flags in 1.628 seconds
2018-11-19 18:13:45 [DEBUG   ] Finished updating ant stats in 1.393 seconds
2018-11-19 18:14:05 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-19 18:14:25 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:14:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:14:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:14:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:14:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:14:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:14:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:14:32 [DEBUG   ] Finished updating board flags in 1.636 seconds
2018-11-19 18:14:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:14:57 [DEBUG   ] Got command RPT from MCS: ref #560920
2018-11-19 18:14:57 [DEBUG   ] summary = ERROR
2018-11-19 18:14:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560920  13 58441 65697377 A  ERRORERROR'
2018-11-19 18:14:57 [DEBUG   ] Got command RPT from MCS: ref #560921
2018-11-19 18:14:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:14:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560921  77 58441 65697581 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:14:57 [DEBUG   ] Got command RPT from MCS: ref #560922
2018-11-19 18:14:57 [DEBUG   ] summary = ERROR
2018-11-19 18:14:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560922  13 58441 65697785 A  ERRORERROR'
2018-11-19 18:14:57 [DEBUG   ] Got command RPT from MCS: ref #560923
2018-11-19 18:14:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:14:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560923  77 58441 65697977 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:14:58 [DEBUG   ] Got command RPT from MCS: ref #560924
2018-11-19 18:14:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:14:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560924  12 58441 65698181 A  ERRORB™™š'
2018-11-19 18:15:05 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:15:25 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:15:31 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 18:15:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:15:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:15:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:15:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:15:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:15:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:15:32 [DEBUG   ] Finished updating board flags in 2.324 seconds
2018-11-19 18:15:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:15:46 [DEBUG   ] Got command RPT from MCS: ref #560925
2018-11-19 18:15:46 [DEBUG   ] summary = ERROR
2018-11-19 18:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560925  13 58441 65746457 A  ERRORERROR'
2018-11-19 18:15:46 [DEBUG   ] Got command RPT from MCS: ref #560926
2018-11-19 18:15:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560926  77 58441 65746458 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:16:05 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:16:25 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 18:16:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:16:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:16:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:16:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:16:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:16:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:16:32 [DEBUG   ] Finished updating board flags in 2.190 seconds
2018-11-19 18:16:45 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:17:05 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:17:25 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:17:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:17:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:17:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:17:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:17:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:17:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:17:32 [DEBUG   ] Finished updating board flags in 1.876 seconds
2018-11-19 18:17:45 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 18:18:05 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:18:25 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 18:18:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:18:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:18:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:18:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:18:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:18:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:18:32 [DEBUG   ] Finished updating board flags in 1.814 seconds
2018-11-19 18:18:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:19:05 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:19:25 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:19:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:19:32 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 18:19:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:19:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:19:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:19:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:19:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:19:32 [DEBUG   ] Finished updating board flags in 1.810 seconds
2018-11-19 18:19:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:19:57 [DEBUG   ] Got command RPT from MCS: ref #560952
2018-11-19 18:19:57 [DEBUG   ] summary = ERROR
2018-11-19 18:19:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560952  13 58441 65997461 A  ERRORERROR'
2018-11-19 18:19:57 [DEBUG   ] Got command RPT from MCS: ref #560953
2018-11-19 18:19:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:19:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560953  77 58441 65997665 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:19:57 [DEBUG   ] Got command RPT from MCS: ref #560954
2018-11-19 18:19:57 [DEBUG   ] summary = ERROR
2018-11-19 18:19:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560954  13 58441 65997869 A  ERRORERROR'
2018-11-19 18:19:58 [DEBUG   ] Got command RPT from MCS: ref #560956
2018-11-19 18:19:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:19:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560956  77 58441 65998061 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:19:58 [DEBUG   ] Got command RPT from MCS: ref #560957
2018-11-19 18:19:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:19:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560957  12 58441 65998265 A  ERRORB™™š'
2018-11-19 18:20:05 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:20:25 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-19 18:20:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:20:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:20:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:20:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:20:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:20:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:20:32 [DEBUG   ] Finished updating board flags in 1.780 seconds
2018-11-19 18:20:45 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:20:46 [DEBUG   ] Got command RPT from MCS: ref #560962
2018-11-19 18:20:46 [DEBUG   ] summary = ERROR
2018-11-19 18:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560962  13 58441 66046461 A  ERRORERROR'
2018-11-19 18:20:46 [DEBUG   ] Got command RPT from MCS: ref #560963
2018-11-19 18:20:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560963  77 58441 66046462 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:21:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 18:21:25 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:21:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 18:21:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:21:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:21:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:21:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:21:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:21:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:21:32 [DEBUG   ] Finished updating board flags in 1.614 seconds
2018-11-19 18:21:45 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 18:22:05 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:22:25 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 18:22:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:22:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:22:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:22:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:22:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:22:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:22:32 [DEBUG   ] Finished updating board flags in 1.650 seconds
2018-11-19 18:22:45 [DEBUG   ] Finished updating ant stats in 0.996 seconds
2018-11-19 18:23:05 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:23:25 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:23:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:23:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:23:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:23:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:23:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:23:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:23:32 [DEBUG   ] Finished updating board flags in 1.539 seconds
2018-11-19 18:23:45 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 18:24:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 18:24:25 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-19 18:24:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:24:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:24:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:24:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:24:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:24:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:24:32 [DEBUG   ] Finished updating board flags in 1.459 seconds
2018-11-19 18:24:45 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 18:24:57 [DEBUG   ] Got command RPT from MCS: ref #560983
2018-11-19 18:24:57 [DEBUG   ] summary = ERROR
2018-11-19 18:24:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560983  13 58441 66297549 A  ERRORERROR'
2018-11-19 18:24:57 [DEBUG   ] Got command RPT from MCS: ref #560984
2018-11-19 18:24:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:24:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560984  77 58441 66297753 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:24:57 [DEBUG   ] Got command RPT from MCS: ref #560985
2018-11-19 18:24:57 [DEBUG   ] summary = ERROR
2018-11-19 18:24:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560985  13 58441 66297945 A  ERRORERROR'
2018-11-19 18:24:58 [DEBUG   ] Got command RPT from MCS: ref #560986
2018-11-19 18:24:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:24:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560986  77 58441 66298149 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:24:58 [DEBUG   ] Got command RPT from MCS: ref #560987
2018-11-19 18:24:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:24:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560987  12 58441 66298353 A  ERRORB˜™š'
2018-11-19 18:25:05 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-19 18:25:26 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-19 18:25:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:25:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:25:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:25:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:25:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:25:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:25:32 [DEBUG   ] Finished updating board flags in 1.471 seconds
2018-11-19 18:25:46 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 18:25:46 [DEBUG   ] Got command RPT from MCS: ref #560992
2018-11-19 18:25:46 [DEBUG   ] summary = ERROR
2018-11-19 18:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560992  13 58441 66346461 A  ERRORERROR'
2018-11-19 18:25:46 [DEBUG   ] Got command RPT from MCS: ref #560993
2018-11-19 18:25:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   560993  77 58441 66346462 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:26:06 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:26:26 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:26:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:26:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:26:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:26:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:26:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:26:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:26:32 [DEBUG   ] Finished updating board flags in 1.469 seconds
2018-11-19 18:26:46 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 18:27:06 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:27:26 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:27:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:27:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:27:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:27:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:27:32 [WARNING ] (13, 1, 'bot')
2018-11-19 18:27:32 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:27:32 [DEBUG   ] Finished updating board flags in 1.532 seconds
2018-11-19 18:27:46 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:28:06 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 18:28:26 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 18:28:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:28:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 18:28:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:28:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:28:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:28:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:28:33 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:28:33 [DEBUG   ] Finished updating board flags in 2.153 seconds
2018-11-19 18:28:46 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:29:06 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 18:29:26 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 18:29:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:29:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:29:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:29:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:29:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:29:33 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:29:33 [DEBUG   ] Finished updating board flags in 1.927 seconds
2018-11-19 18:29:46 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 18:29:57 [DEBUG   ] Got command RPT from MCS: ref #561015
2018-11-19 18:29:57 [DEBUG   ] summary = ERROR
2018-11-19 18:29:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561015  13 58441 66597629 A  ERRORERROR'
2018-11-19 18:29:57 [DEBUG   ] Got command RPT from MCS: ref #561016
2018-11-19 18:29:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:29:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561016  77 58441 66597833 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:29:58 [DEBUG   ] Got command RPT from MCS: ref #561017
2018-11-19 18:29:58 [DEBUG   ] summary = ERROR
2018-11-19 18:29:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561017  13 58441 66598037 A  ERRORERROR'
2018-11-19 18:29:58 [DEBUG   ] Got command RPT from MCS: ref #561019
2018-11-19 18:29:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:29:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561019  77 58441 66598241 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:29:58 [DEBUG   ] Got command RPT from MCS: ref #561020
2018-11-19 18:29:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:29:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561020  12 58441 66598433 A  ERRORB˜™š'
2018-11-19 18:30:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 18:30:26 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:30:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:30:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:30:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:30:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:30:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:30:33 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:30:33 [DEBUG   ] Finished updating board flags in 1.809 seconds
2018-11-19 18:30:46 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 18:30:46 [DEBUG   ] Got command RPT from MCS: ref #561025
2018-11-19 18:30:46 [DEBUG   ] summary = ERROR
2018-11-19 18:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561025  13 58441 66646473 A  ERRORERROR'
2018-11-19 18:30:46 [DEBUG   ] Got command RPT from MCS: ref #561026
2018-11-19 18:30:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561026  77 58441 66646474 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:31:06 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 18:31:26 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 18:31:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:31:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:31:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:31:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:31:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:31:33 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:31:33 [DEBUG   ] Finished updating board flags in 1.838 seconds
2018-11-19 18:31:46 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 18:32:06 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 18:32:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:32:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:32:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:32:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:32:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:32:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:32:33 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:32:33 [DEBUG   ] Finished updating board flags in 1.761 seconds
2018-11-19 18:32:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:33:06 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 18:33:26 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 18:33:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:33:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:33:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:33:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:33:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:33:33 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:33:33 [DEBUG   ] Finished updating board flags in 1.602 seconds
2018-11-19 18:33:46 [DEBUG   ] Finished updating ant stats in 0.797 seconds
2018-11-19 18:34:06 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:34:26 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 18:34:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:34:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:34:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:34:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:34:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:34:33 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:34:33 [DEBUG   ] Finished updating board flags in 1.590 seconds
2018-11-19 18:34:46 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 18:34:57 [DEBUG   ] Got command RPT from MCS: ref #561050
2018-11-19 18:34:57 [DEBUG   ] summary = ERROR
2018-11-19 18:34:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561050  13 58441 66897717 A  ERRORERROR'
2018-11-19 18:34:57 [DEBUG   ] Got command RPT from MCS: ref #561051
2018-11-19 18:34:57 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:34:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561051  77 58441 66897921 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:34:58 [DEBUG   ] Got command RPT from MCS: ref #561052
2018-11-19 18:34:58 [DEBUG   ] summary = ERROR
2018-11-19 18:34:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561052  13 58441 66898125 A  ERRORERROR'
2018-11-19 18:34:58 [DEBUG   ] Got command RPT from MCS: ref #561053
2018-11-19 18:34:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:34:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561053  77 58441 66898317 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:34:58 [DEBUG   ] Got command RPT from MCS: ref #561054
2018-11-19 18:34:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:34:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561054  12 58441 66898521 A  ERRORB—™š'
2018-11-19 18:35:06 [DEBUG   ] Finished updating ant stats in 0.597 seconds
2018-11-19 18:35:26 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 18:35:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:35:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:35:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:35:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:35:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:35:33 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:35:33 [DEBUG   ] Finished updating board flags in 1.418 seconds
2018-11-19 18:35:46 [DEBUG   ] Got command RPT from MCS: ref #561055
2018-11-19 18:35:46 [DEBUG   ] summary = ERROR
2018-11-19 18:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561055  13 58441 66946469 A  ERRORERROR'
2018-11-19 18:35:46 [DEBUG   ] Got command RPT from MCS: ref #561056
2018-11-19 18:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561056  77 58441 66946470 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:35:47 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-19 18:36:07 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 18:36:27 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 18:36:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:36:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:36:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:36:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:36:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:36:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:36:34 [DEBUG   ] Finished updating board flags in 2.392 seconds
2018-11-19 18:36:47 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 18:37:07 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 18:37:27 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 18:37:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:37:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 18:37:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:37:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:37:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:37:33 [WARNING ] (13, 1, 'bot')
2018-11-19 18:37:33 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:37:33 [DEBUG   ] Finished updating board flags in 1.316 seconds
2018-11-19 18:37:47 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:38:07 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 18:38:27 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 18:38:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:38:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:38:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:38:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:38:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:38:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:38:34 [DEBUG   ] Finished updating board flags in 2.124 seconds
2018-11-19 18:38:47 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:39:07 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-19 18:39:27 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 18:39:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 18:39:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:39:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:39:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:39:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:39:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:39:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:39:34 [DEBUG   ] Finished updating board flags in 1.958 seconds
2018-11-19 18:39:47 [DEBUG   ] Finished updating ant stats in 0.595 seconds
2018-11-19 18:39:57 [DEBUG   ] Got command RPT from MCS: ref #561078
2018-11-19 18:39:57 [DEBUG   ] summary = ERROR
2018-11-19 18:39:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561078  13 58441 67197801 A  ERRORERROR'
2018-11-19 18:39:58 [DEBUG   ] Got command RPT from MCS: ref #561079
2018-11-19 18:39:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:39:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561079  77 58441 67198005 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:39:58 [DEBUG   ] Got command RPT from MCS: ref #561080
2018-11-19 18:39:58 [DEBUG   ] summary = ERROR
2018-11-19 18:39:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561080  13 58441 67198209 A  ERRORERROR'
2018-11-19 18:39:58 [DEBUG   ] Got command RPT from MCS: ref #561082
2018-11-19 18:39:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:39:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561082  77 58441 67198401 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:39:58 [DEBUG   ] Got command RPT from MCS: ref #561083
2018-11-19 18:39:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:39:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561083  12 58441 67198605 A  ERRORB—™š'
2018-11-19 18:40:08 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-19 18:40:28 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-19 18:40:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:40:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 18:40:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:40:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:40:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:40:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:40:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:40:34 [DEBUG   ] Finished updating board flags in 1.823 seconds
2018-11-19 18:40:46 [DEBUG   ] Got command RPT from MCS: ref #561092
2018-11-19 18:40:46 [DEBUG   ] summary = ERROR
2018-11-19 18:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561092  13 58441 67246473 A  ERRORERROR'
2018-11-19 18:40:46 [DEBUG   ] Got command RPT from MCS: ref #561093
2018-11-19 18:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561093  77 58441 67246474 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:40:48 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 18:41:08 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 18:41:28 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 18:41:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:41:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:41:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:41:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:41:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:41:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:41:34 [DEBUG   ] Finished updating board flags in 1.782 seconds
2018-11-19 18:41:48 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-19 18:42:08 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 18:42:28 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 18:42:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:42:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:42:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:42:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:42:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:42:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:42:34 [DEBUG   ] Finished updating board flags in 1.615 seconds
2018-11-19 18:42:48 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:43:08 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 18:43:28 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:43:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 18:43:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:43:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:43:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:43:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:43:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:43:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:43:34 [DEBUG   ] Finished updating board flags in 1.637 seconds
2018-11-19 18:43:48 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:44:08 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:44:28 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 18:44:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 18:44:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:44:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 18:44:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 18:44:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:44:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:44:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:44:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:44:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:44:34 [DEBUG   ] Finished updating board flags in 1.560 seconds
2018-11-19 18:44:48 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-19 18:44:57 [DEBUG   ] Got command RPT from MCS: ref #561113
2018-11-19 18:44:57 [DEBUG   ] summary = ERROR
2018-11-19 18:44:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561113  13 58441 67497889 A  ERRORERROR'
2018-11-19 18:44:58 [DEBUG   ] Got command RPT from MCS: ref #561114
2018-11-19 18:44:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:44:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561114  77 58441 67498093 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:44:58 [DEBUG   ] Got command RPT from MCS: ref #561115
2018-11-19 18:44:58 [DEBUG   ] summary = ERROR
2018-11-19 18:44:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561115  13 58441 67498285 A  ERRORERROR'
2018-11-19 18:44:58 [DEBUG   ] Got command RPT from MCS: ref #561116
2018-11-19 18:44:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:44:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561116  77 58441 67498489 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:44:58 [DEBUG   ] Got command RPT from MCS: ref #561117
2018-11-19 18:44:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:44:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561117  12 58441 67498693 A  ERRORB™™š'
2018-11-19 18:45:08 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:45:28 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:45:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:45:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 18:45:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:45:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:45:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:45:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:45:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:45:34 [DEBUG   ] Finished updating board flags in 1.434 seconds
2018-11-19 18:45:46 [DEBUG   ] Got command RPT from MCS: ref #561118
2018-11-19 18:45:46 [DEBUG   ] summary = ERROR
2018-11-19 18:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561118  13 58441 67546473 A  ERRORERROR'
2018-11-19 18:45:46 [DEBUG   ] Got command RPT from MCS: ref #561119
2018-11-19 18:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561119  77 58441 67546474 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:45:48 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 18:46:08 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 18:46:28 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 18:46:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:46:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:46:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:46:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:46:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:46:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:46:34 [DEBUG   ] Finished updating board flags in 1.455 seconds
2018-11-19 18:46:49 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-19 18:47:09 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 18:47:29 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:47:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 18:47:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:47:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:47:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:47:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:47:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:47:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:47:34 [DEBUG   ] Finished updating board flags in 1.427 seconds
2018-11-19 18:47:49 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 18:48:09 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-19 18:48:29 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 18:48:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 18:48:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:48:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:48:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:48:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:48:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:48:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:48:34 [DEBUG   ] Finished updating board flags in 1.472 seconds
2018-11-19 18:48:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 18:49:09 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:49:29 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 18:49:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 18:49:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:49:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:49:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:49:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:49:34 [WARNING ] (13, 1, 'bot')
2018-11-19 18:49:34 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:49:34 [DEBUG   ] Finished updating board flags in 1.469 seconds
2018-11-19 18:49:49 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:49:57 [DEBUG   ] Got command RPT from MCS: ref #561145
2018-11-19 18:49:57 [DEBUG   ] summary = ERROR
2018-11-19 18:49:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561145  13 58441 67797977 A  ERRORERROR'
2018-11-19 18:49:58 [DEBUG   ] Got command RPT from MCS: ref #561146
2018-11-19 18:49:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:49:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561146  77 58441 67798169 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:49:58 [DEBUG   ] Got command RPT from MCS: ref #561147
2018-11-19 18:49:58 [DEBUG   ] summary = ERROR
2018-11-19 18:49:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561147  13 58441 67798373 A  ERRORERROR'
2018-11-19 18:49:58 [DEBUG   ] Got command RPT from MCS: ref #561149
2018-11-19 18:49:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:49:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561149  77 58441 67798577 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:49:58 [DEBUG   ] Got command RPT from MCS: ref #561150
2018-11-19 18:49:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:49:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561150  12 58441 67798781 A  ERRORB™™š'
2018-11-19 18:50:09 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 18:50:29 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 18:50:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:50:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:50:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:50:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:50:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:50:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:50:35 [DEBUG   ] Finished updating board flags in 2.236 seconds
2018-11-19 18:50:46 [DEBUG   ] Got command RPT from MCS: ref #561155
2018-11-19 18:50:46 [DEBUG   ] summary = ERROR
2018-11-19 18:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561155  13 58441 67846469 A  ERRORERROR'
2018-11-19 18:50:46 [DEBUG   ] Got command RPT from MCS: ref #561156
2018-11-19 18:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561156  77 58441 67846470 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:50:49 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:51:09 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:51:29 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 18:51:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:51:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:51:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:51:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:51:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:51:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:51:35 [DEBUG   ] Finished updating board flags in 2.223 seconds
2018-11-19 18:51:49 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 18:52:09 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 18:52:29 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 18:52:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:52:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:52:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:52:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:52:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:52:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:52:35 [DEBUG   ] Finished updating board flags in 2.143 seconds
2018-11-19 18:52:49 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:53:09 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:53:29 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:53:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:53:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:53:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:53:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:53:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:53:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:53:35 [DEBUG   ] Finished updating board flags in 2.022 seconds
2018-11-19 18:53:49 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:54:09 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:54:29 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 18:54:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:54:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:54:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:54:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:54:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:54:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:54:35 [DEBUG   ] Finished updating board flags in 1.949 seconds
2018-11-19 18:54:49 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 18:54:58 [DEBUG   ] Got command RPT from MCS: ref #561176
2018-11-19 18:54:58 [DEBUG   ] summary = ERROR
2018-11-19 18:54:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561176  13 58441 68098061 A  ERRORERROR'
2018-11-19 18:54:58 [DEBUG   ] Got command RPT from MCS: ref #561177
2018-11-19 18:54:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:54:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561177  77 58441 68098265 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:54:58 [DEBUG   ] Got command RPT from MCS: ref #561178
2018-11-19 18:54:58 [DEBUG   ] summary = ERROR
2018-11-19 18:54:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561178  13 58441 68098457 A  ERRORERROR'
2018-11-19 18:54:58 [DEBUG   ] Got command RPT from MCS: ref #561179
2018-11-19 18:54:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:54:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561179  77 58441 68098662 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:54:58 [DEBUG   ] Got command RPT from MCS: ref #561180
2018-11-19 18:54:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:54:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561180  12 58441 68098865 A  ERRORB™™š'
2018-11-19 18:55:09 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:55:29 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:55:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:55:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:55:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:55:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:55:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:55:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:55:35 [DEBUG   ] Finished updating board flags in 1.825 seconds
2018-11-19 18:55:46 [DEBUG   ] Got command RPT from MCS: ref #561185
2018-11-19 18:55:46 [DEBUG   ] summary = ERROR
2018-11-19 18:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561185  13 58441 68146469 A  ERRORERROR'
2018-11-19 18:55:46 [DEBUG   ] Got command RPT from MCS: ref #561186
2018-11-19 18:55:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561186  77 58441 68146470 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:55:49 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:56:09 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 18:56:29 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:56:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:56:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:56:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:56:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:56:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:56:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:56:35 [DEBUG   ] Finished updating board flags in 1.830 seconds
2018-11-19 18:56:49 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 18:57:09 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 18:57:29 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 18:57:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:57:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:57:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:57:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:57:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:57:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:57:35 [DEBUG   ] Finished updating board flags in 1.824 seconds
2018-11-19 18:57:49 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-19 18:58:09 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:58:29 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:58:34 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 18:58:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:58:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:58:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:58:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:58:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:58:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:58:35 [DEBUG   ] Finished updating board flags in 1.720 seconds
2018-11-19 18:58:49 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 18:59:09 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 18:59:29 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 18:59:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 18:59:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:59:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:59:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:59:35 [WARNING ] (13, 1, 'bot')
2018-11-19 18:59:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 18:59:35 [DEBUG   ] Finished updating board flags in 1.670 seconds
2018-11-19 18:59:49 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 18:59:58 [DEBUG   ] Got command RPT from MCS: ref #561208
2018-11-19 18:59:58 [DEBUG   ] summary = ERROR
2018-11-19 18:59:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561208  13 58441 68398145 A  ERRORERROR'
2018-11-19 18:59:58 [DEBUG   ] Got command RPT from MCS: ref #561209
2018-11-19 18:59:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:59:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561209  77 58441 68398349 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:59:58 [DEBUG   ] Got command RPT from MCS: ref #561210
2018-11-19 18:59:58 [DEBUG   ] summary = ERROR
2018-11-19 18:59:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561210  13 58441 68398553 A  ERRORERROR'
2018-11-19 18:59:58 [DEBUG   ] Got command RPT from MCS: ref #561212
2018-11-19 18:59:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 18:59:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561212  77 58441 68398745 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 18:59:58 [DEBUG   ] Got command RPT from MCS: ref #561213
2018-11-19 18:59:58 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 18:59:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561213  12 58441 68398953 A  ERRORB˜™š'
2018-11-19 19:00:09 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 19:00:29 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:00:34 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:00:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:00:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 19:00:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:00:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:00:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:00:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:00:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:00:35 [DEBUG   ] Finished updating board flags in 1.608 seconds
2018-11-19 19:00:46 [DEBUG   ] Got command RPT from MCS: ref #561218
2018-11-19 19:00:46 [DEBUG   ] summary = ERROR
2018-11-19 19:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561218  13 58441 68446473 A  ERRORERROR'
2018-11-19 19:00:46 [DEBUG   ] Got command RPT from MCS: ref #561219
2018-11-19 19:00:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561219  77 58441 68446474 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:00:49 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 19:01:10 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-19 19:01:29 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:01:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:01:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:01:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:01:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:01:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:01:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:01:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:01:35 [DEBUG   ] Finished updating board flags in 1.525 seconds
2018-11-19 19:01:50 [DEBUG   ] Finished updating ant stats in 1.394 seconds
2018-11-19 19:02:10 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 19:02:30 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 19:02:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:02:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:02:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:02:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:02:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:02:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:02:35 [DEBUG   ] Finished updating board flags in 1.424 seconds
2018-11-19 19:02:50 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 19:03:10 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:03:30 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 19:03:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:03:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:03:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:03:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:03:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:03:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:03:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:03:35 [DEBUG   ] Finished updating board flags in 1.383 seconds
2018-11-19 19:03:50 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 19:04:10 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 19:04:30 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:04:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:04:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 19:04:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:04:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:04:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:04:35 [WARNING ] (13, 1, 'bot')
2018-11-19 19:04:35 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:04:35 [DEBUG   ] Finished updating board flags in 1.370 seconds
2018-11-19 19:04:50 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:04:58 [DEBUG   ] Got command RPT from MCS: ref #561243
2018-11-19 19:04:58 [DEBUG   ] summary = ERROR
2018-11-19 19:04:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561243  13 58441 68698233 A  ERRORERROR'
2018-11-19 19:04:58 [DEBUG   ] Got command RPT from MCS: ref #561244
2018-11-19 19:04:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:04:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561244  77 58441 68698425 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:04:58 [DEBUG   ] Got command RPT from MCS: ref #561245
2018-11-19 19:04:58 [DEBUG   ] summary = ERROR
2018-11-19 19:04:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561245  13 58441 68698629 A  ERRORERROR'
2018-11-19 19:04:58 [DEBUG   ] Got command RPT from MCS: ref #561246
2018-11-19 19:04:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:04:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561246  77 58441 68698833 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:04:59 [DEBUG   ] Got command RPT from MCS: ref #561247
2018-11-19 19:04:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:04:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561247  12 58441 68699037 A  ERRORB˜™š'
2018-11-19 19:05:10 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 19:05:30 [DEBUG   ] Finished updating ant stats in 0.797 seconds
2018-11-19 19:05:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:05:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:05:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:05:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:05:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:05:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:05:36 [DEBUG   ] Finished updating board flags in 2.048 seconds
2018-11-19 19:05:46 [DEBUG   ] Got command RPT from MCS: ref #561248
2018-11-19 19:05:46 [DEBUG   ] summary = ERROR
2018-11-19 19:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561248  13 58441 68746473 A  ERRORERROR'
2018-11-19 19:05:46 [DEBUG   ] Got command RPT from MCS: ref #561249
2018-11-19 19:05:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561249  77 58441 68746474 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:05:50 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 19:06:10 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:06:31 [DEBUG   ] Finished updating ant stats in 1.395 seconds
2018-11-19 19:06:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:06:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:06:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:06:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:06:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:06:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:06:36 [DEBUG   ] Finished updating board flags in 2.008 seconds
2018-11-19 19:06:51 [DEBUG   ] Finished updating ant stats in 1.212 seconds
2018-11-19 19:07:11 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-19 19:07:31 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 19:07:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:07:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:07:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:07:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:07:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:07:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:07:36 [DEBUG   ] Finished updating board flags in 1.987 seconds
2018-11-19 19:07:51 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 19:08:11 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:08:31 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 19:08:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:08:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:08:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:08:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:08:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:08:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:08:36 [DEBUG   ] Finished updating board flags in 1.822 seconds
2018-11-19 19:08:51 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:09:11 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:09:31 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:09:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:09:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:09:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:09:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:09:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:09:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:09:36 [DEBUG   ] Finished updating board flags in 1.782 seconds
2018-11-19 19:09:51 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 19:09:58 [DEBUG   ] Got command RPT from MCS: ref #561271
2018-11-19 19:09:58 [DEBUG   ] summary = ERROR
2018-11-19 19:09:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561271  13 58441 68998321 A  ERRORERROR'
2018-11-19 19:09:58 [DEBUG   ] Got command RPT from MCS: ref #561272
2018-11-19 19:09:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:09:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561272  77 58441 68998513 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:09:58 [DEBUG   ] Got command RPT from MCS: ref #561273
2018-11-19 19:09:58 [DEBUG   ] summary = ERROR
2018-11-19 19:09:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561273  13 58441 68998717 A  ERRORERROR'
2018-11-19 19:09:58 [DEBUG   ] Got command RPT from MCS: ref #561275
2018-11-19 19:09:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:09:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561275  77 58441 68998921 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:09:59 [DEBUG   ] Got command RPT from MCS: ref #561277
2018-11-19 19:09:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:09:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561277  12 58441 68999125 A  ERRORB˜™š'
2018-11-19 19:10:11 [DEBUG   ] Finished updating ant stats in 0.995 seconds
2018-11-19 19:10:31 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:10:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:10:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:10:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:10:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:10:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:10:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:10:36 [DEBUG   ] Finished updating board flags in 1.612 seconds
2018-11-19 19:10:46 [DEBUG   ] Got command RPT from MCS: ref #561285
2018-11-19 19:10:46 [DEBUG   ] summary = ERROR
2018-11-19 19:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561285  13 58441 69046473 A  ERRORERROR'
2018-11-19 19:10:46 [DEBUG   ] Got command RPT from MCS: ref #561286
2018-11-19 19:10:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561286  77 58441 69046474 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:10:51 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:11:11 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 19:11:31 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 19:11:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:11:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:11:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:11:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:11:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:11:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:11:36 [DEBUG   ] Finished updating board flags in 1.633 seconds
2018-11-19 19:11:51 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:12:11 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:12:31 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 19:12:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:12:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:12:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:12:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:12:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:12:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:12:36 [DEBUG   ] Finished updating board flags in 1.556 seconds
2018-11-19 19:12:51 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 19:13:11 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:13:32 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-19 19:13:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:13:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:13:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:13:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:13:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:13:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:13:36 [DEBUG   ] Finished updating board flags in 1.425 seconds
2018-11-19 19:13:52 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-19 19:14:12 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 19:14:32 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 19:14:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 19:14:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:14:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:14:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:14:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:14:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:14:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:14:36 [DEBUG   ] Finished updating board flags in 1.383 seconds
2018-11-19 19:14:52 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-19 19:14:58 [DEBUG   ] Got command RPT from MCS: ref #561306
2018-11-19 19:14:58 [DEBUG   ] summary = ERROR
2018-11-19 19:14:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561306  13 58441 69298397 A  ERRORERROR'
2018-11-19 19:14:58 [DEBUG   ] Got command RPT from MCS: ref #561307
2018-11-19 19:14:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:14:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561307  77 58441 69298601 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:14:58 [DEBUG   ] Got command RPT from MCS: ref #561308
2018-11-19 19:14:58 [DEBUG   ] summary = ERROR
2018-11-19 19:14:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561308  13 58441 69298805 A  ERRORERROR'
2018-11-19 19:14:59 [DEBUG   ] Got command RPT from MCS: ref #561309
2018-11-19 19:14:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:14:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561309  77 58441 69299009 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:14:59 [DEBUG   ] Got command RPT from MCS: ref #561310
2018-11-19 19:14:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:14:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561310  12 58441 69299201 A  ERRORB˜™š'
2018-11-19 19:15:12 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:15:32 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:15:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:15:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 19:15:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:15:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:15:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:15:36 [WARNING ] (13, 1, 'bot')
2018-11-19 19:15:36 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:15:36 [DEBUG   ] Finished updating board flags in 1.377 seconds
2018-11-19 19:15:46 [DEBUG   ] Got command RPT from MCS: ref #561311
2018-11-19 19:15:46 [DEBUG   ] summary = ERROR
2018-11-19 19:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561311  13 58441 69346481 A  ERRORERROR'
2018-11-19 19:15:46 [DEBUG   ] Got command RPT from MCS: ref #561312
2018-11-19 19:15:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561312  77 58441 69346482 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:15:52 [DEBUG   ] Finished updating ant stats in 1.005 seconds
2018-11-19 19:16:00 [DEBUG   ] Got command RPT from MCS: ref #561328
2018-11-19 19:16:00 [DEBUG   ] summary = ERROR
2018-11-19 19:16:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561328  13 58441 69360477 A  ERRORERROR'
2018-11-19 19:16:00 [DEBUG   ] Got command RPT from MCS: ref #561329
2018-11-19 19:16:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:16:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561329  77 58441 69360478 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:16:05 [DEBUG   ] Got command INI from MCS: ref #561334
2018-11-19 19:16:05 [INFO    ] Parsing config file '/lwa/software/defaults.cfg'
2018-11-19 19:16:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_INI   561334   8 58441 69365486 ABOOTING'
2018-11-19 19:16:29 [INFO    ] -------------------
2018-11-19 19:16:29 [INFO    ]  LWA board status
2018-11-19 19:16:29 [INFO    ] -------------------
2018-11-19 19:16:29 [INFO    ] DP 1: 2 lwa27
2018-11-19 19:16:29 [INFO    ] DP 1: 3 lwa24
2018-11-19 19:16:29 [INFO    ] DP 1: 4 lwa23
2018-11-19 19:16:29 [INFO    ] DP 1: 5 lwa18
2018-11-19 19:16:29 [INFO    ] DP 1: 6 lwa10
2018-11-19 19:16:29 [INFO    ] DP 1: 7 lwa32
2018-11-19 19:16:29 [INFO    ] DP 1: 8 lwa11
2018-11-19 19:16:29 [INFO    ] DP 1: 9 lwa30
2018-11-19 19:16:29 [INFO    ] DP 1: 10 lwa12
2018-11-19 19:16:29 [INFO    ] DP 1: 11 lwa8
2018-11-19 19:16:29 [INFO    ] DP 1: 12 lwa13
2018-11-19 19:16:29 [INFO    ] DP 1: 13 lwa26
2018-11-19 19:16:29 [INFO    ] DP 1: 14 lwa14
2018-11-19 19:16:29 [INFO    ] DP 1: 16 lwa22
2018-11-19 19:16:29 [INFO    ] DP 1: 17 lwa20
2018-11-19 19:16:29 [INFO    ] DP 1: 18 lwa1
2018-11-19 19:16:29 [INFO    ] DP 1: 19 lwa5
2018-11-19 19:16:29 [INFO    ] DP 1: 20 lwa25
2018-11-19 19:16:29 [INFO    ] DP 1: 21 lwa35
2018-11-19 19:16:29 [INFO    ] DP 1: 22 lwa19
2018-11-19 19:16:29 [INFO    ] DP 1: 23 lwa21
2018-11-19 19:16:29 [INFO    ] DP 1: 24 lwa6
2018-11-19 19:16:29 [INFO    ] DP 1: 25 lwa7
2018-11-19 19:16:29 [INFO    ] DP 1: 26 lwa33
2018-11-19 19:16:29 [INFO    ] DP 1: 27 lwa15
2018-11-19 19:16:29 [INFO    ] DP 1: 28 lwa28
2018-11-19 19:16:29 [INFO    ] DP 2: 1 lwa16
2018-11-19 19:16:29 [INFO    ] DP 2: 15 lwa34
2018-11-19 19:16:29 [INFO    ] Total of 28 boards
2018-11-19 19:16:29 [INFO    ] Total of 260 stands
2018-11-19 19:16:29 [INFO    ] -------------------
2018-11-19 19:16:29 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:29 [DEBUG   ] sendHostnameCommand - Sent hostname lwa27 command /usr/local/bin/lwa27 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:29 [DEBUG   ] sendHostnameCommand - Sent hostname lwa24 command /usr/local/bin/lwa24 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:30 [DEBUG   ] sendHostnameCommand - Sent hostname lwa23 command /usr/local/bin/lwa23 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:30 [DEBUG   ] sendHostnameCommand - Sent hostname lwa18 command /usr/local/bin/lwa18 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:30 [DEBUG   ] sendHostnameCommand - Sent hostname lwa10 command /usr/local/bin/lwa10 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:30 [DEBUG   ] sendHostnameCommand - Sent hostname lwa32 command /usr/local/bin/lwa32 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:31 [DEBUG   ] sendHostnameCommand - Sent hostname lwa11 command /usr/local/bin/lwa11 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:31 [DEBUG   ] sendHostnameCommand - Sent hostname lwa30 command /usr/local/bin/lwa30 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:31 [DEBUG   ] sendHostnameCommand - Sent hostname lwa12 command /usr/local/bin/lwa12 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:31 [DEBUG   ] sendHostnameCommand - Sent hostname lwa8 command /usr/local/bin/lwa8 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:32 [DEBUG   ] sendHostnameCommand - Sent hostname lwa13 command /usr/local/bin/lwa13 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:32 [DEBUG   ] sendHostnameCommand - Sent hostname lwa26 command /usr/local/bin/lwa26 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:32 [DEBUG   ] sendHostnameCommand - Sent hostname lwa14 command /usr/local/bin/lwa14 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:32 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:33 [DEBUG   ] sendHostnameCommand - Sent hostname lwa22 command /usr/local/bin/lwa22 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:33 [DEBUG   ] sendHostnameCommand - Sent hostname lwa20 command /usr/local/bin/lwa20 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:33 [DEBUG   ] sendHostnameCommand - Sent hostname lwa1 command /usr/local/bin/lwa1 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:34 [DEBUG   ] sendHostnameCommand - Sent hostname lwa5 command /usr/local/bin/lwa5 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:34 [DEBUG   ] sendHostnameCommand - Sent hostname lwa25 command /usr/local/bin/lwa25 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:34 [DEBUG   ] sendHostnameCommand - Sent hostname lwa35 command /usr/local/bin/lwa35 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:34 [DEBUG   ] sendHostnameCommand - Sent hostname lwa19 command /usr/local/bin/lwa19 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:35 [DEBUG   ] sendHostnameCommand - Sent hostname lwa21 command /usr/local/bin/lwa21 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:35 [DEBUG   ] sendHostnameCommand - Sent hostname lwa6 command /usr/local/bin/lwa6 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:35 [DEBUG   ] sendHostnameCommand - Sent hostname lwa7 command /usr/local/bin/lwa7 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:35 [DEBUG   ] sendHostnameCommand - Sent hostname lwa33 command /usr/local/bin/lwa33 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:36 [DEBUG   ] sendHostnameCommand - Sent hostname lwa15 command /usr/local/bin/lwa15 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:36 [DEBUG   ] sendHostnameCommand - Sent hostname lwa28 command /usr/local/bin/lwa28 /lwa/software/dp_shutdown.sh
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa27', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2840
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa24', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2841
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa23', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2842
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa18', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2843
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa10', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2844
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa32', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2845
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa11', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2846
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa30', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2847
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa12', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2848
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa8', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2849
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa13', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2850
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa26', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2851
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa14', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2852
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa22', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2853
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa20', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2854
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa1', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2855
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa5', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2856
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa25', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2857
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa35', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2858
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa19', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2859
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa21', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2860
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa6', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2861
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa7', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2862
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa33', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2863
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa15', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2864
2018-11-19 19:16:36 [DEBUG   ] 2018-11-19 19:16:36 Starting ['/usr/local/bin/lwa28', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 2865
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa27', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2840 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa24', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2841 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa23', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2842 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa18', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2843 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa10', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2844 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa32', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2845 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa11', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2846 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa30', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2847 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa12', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2848 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa8', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2849 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa13', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2850 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa26', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2851 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa14', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2852 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa22', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2853 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa20', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2854 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa1', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2855 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa5', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2856 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa25', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2857 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa35', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2858 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa19', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2859 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa21', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2860 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa6', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2861 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa7', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2862 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa33', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2863 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa15', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2864 exited with status 0
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 ['/usr/local/bin/lwa28', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 2865 exited with status 0
2018-11-19 19:16:49 [INFO    ] Finish loading DP1 firmware'/lwa/hardware/lwa_dp_top_2_1_1_49.bin'
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa27', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa24', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa23', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa18', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa10', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa32', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa11', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa30', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa12', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa8', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa13', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa26', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa14', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa22', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa20', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa1', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa5', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa25', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa35', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa19', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa21', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa6', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa7', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa33', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa15', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] spawn_process: at 2018-11-19 19:16:49 running ['/usr/local/bin/lwa28', '/lwa/software/dp1_init.sh']
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 Starting ['/usr/local/bin/lwa16', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] pid 2892
2018-11-19 19:16:49 [DEBUG   ] 2018-11-19 19:16:49 Starting ['/usr/local/bin/lwa34', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] pid 2893
2018-11-19 19:16:54 [DEBUG   ] 2018-11-19 19:16:54 ['/usr/local/bin/lwa16', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] Process 2892 exited with status 0
2018-11-19 19:16:54 [DEBUG   ] 2018-11-19 19:16:54 ['/usr/local/bin/lwa34', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] Process 2893 exited with status 0
2018-11-19 19:16:54 [INFO    ] Finish loading DP2 firmware '/lwa/hardware/lwa_drx_top_3_2_0_6.bin'
2018-11-19 19:16:54 [DEBUG   ] spawn_process: at 2018-11-19 19:16:54 running ['/usr/local/bin/lwa16', '/lwa/software/dp2_init.sh']
2018-11-19 19:16:54 [DEBUG   ] spawn_process: at 2018-11-19 19:16:54 running ['/usr/local/bin/lwa34', '/lwa/software/dp2_init.sh']
2018-11-19 19:16:54 [INFO    ] Setting Beam 1 xbars
2018-11-19 19:16:54 [DEBUG   ] board 11 xbars (bd11 to bd5 through bot5) - beam 1
2018-11-19 19:16:54 [DEBUG   ] board 5 xbars (bd5 from bd11 through bot1) - beam 1
2018-11-19 19:16:54 [DEBUG   ] board 5 xbars (bd5 to bd13 through bot5) - beam 1
2018-11-19 19:16:55 [DEBUG   ] board 13 xbars (bd13 from bd5 through bot1) - beam 1
2018-11-19 19:16:55 [DEBUG   ] board 13 xbars (bd13 to bd7 through bot5) - beam 1
2018-11-19 19:16:55 [DEBUG   ] board 7 xbars (bd7 from bd13 through bot1) - beam 1
2018-11-19 19:16:55 [DEBUG   ] board 7 xbars (bd7 to bd9 through bot5) - beam 1
2018-11-19 19:16:55 [DEBUG   ] board 9 xbars (bd9 from bd7 through bot1) - beam 1
2018-11-19 19:16:55 [DEBUG   ] board 9 xbars (bd9 to bd3 through bot5) - beam 1
2018-11-19 19:16:55 [DEBUG   ] board 3 xbars (bd3 from bd9 through bot1) - beam 1
2018-11-19 19:16:55 [DEBUG   ] board 3 xbars (bd3 to bd2 through bot5) - beam 1
2018-11-19 19:16:55 [DEBUG   ] board 2 xbars (bd2 from bd3 through bot1) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 2 xbars (bd2 to bd6 through bot5) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 6 xbars (bd6 from bd2 through bot1) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 6 xbars (bd6 to bd12 through bot5) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 12 xbars (bd12 from bd6 through bot1) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 12 xbars (bd12 to bd8 through bot5) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 8 xbars (bd8 from bd12 through bot1) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 8 xbars (bd8 to bd4 through bot5) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 4 xbars (bd4 from bd8 through bot1) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 4 xbars (bd4 to bd10 through bot5) - beam 1
2018-11-19 19:16:56 [DEBUG   ] board 10 xbars (bd10 from bd4 through bot1) - beam 1
2018-11-19 19:16:57 [DEBUG   ] board 10 xbars (bd10 to bd14 through bot5) - beam 1
2018-11-19 19:16:57 [DEBUG   ] board 14 xbars (bd14 from bd10 through top1) - beam 1
2018-11-19 19:16:57 [DEBUG   ] board 28 xbars  (bd 28 to bd24 through top1)- beam 1
2018-11-19 19:16:57 [DEBUG   ] board 24 xbars (bd24 from bd28 through bot5) - beam 1
2018-11-19 19:16:57 [DEBUG   ] board 24 xbars (bd24 to bd18 through bot1) - beam 1
2018-11-19 19:16:57 [DEBUG   ] board 18 xbars (bd18 from bd24 through bot5) - beam 1
2018-11-19 19:16:57 [DEBUG   ] board 18 xbars (bd18 to bd22 through bot1) - beam 1
2018-11-19 19:16:57 [DEBUG   ] board 22 xbars (bd22 from bd18 through bot5) - beam 1
2018-11-19 19:16:57 [DEBUG   ] board 22 xbars (bd22 to bd26 through bot1) - beam 1
2018-11-19 19:16:58 [DEBUG   ] board 26 xbars (bd26 from bd22 through bot5) - beam 1
2018-11-19 19:16:58 [DEBUG   ] board 26 xbars (bd26 to bd20 through bot1) - beam 1
2018-11-19 19:16:58 [DEBUG   ] board 20 xbars (bd20 from bd26 through bot5) - beam 1
2018-11-19 19:16:58 [DEBUG   ] board 20 xbars (bd20 to bd16 through bot1) - beam 1
2018-11-19 19:16:58 [DEBUG   ] board 16 xbars (bd16 from bd20 through bot5) - beam 1
2018-11-19 19:16:58 [DEBUG   ] board 16 xbars (bd16 to bd17 through bot1) - beam 1
2018-11-19 19:16:58 [DEBUG   ] board 17 xbars (bd17 from bd16 through bot5) - beam 1
2018-11-19 19:16:58 [DEBUG   ] board 17 xbars (bd17 to bd23 through bot1) - beam 1
2018-11-19 19:16:58 [DEBUG   ] board 23 xbars (bd23 from bd17 through bot5) - beam 1
2018-11-19 19:16:59 [DEBUG   ] board 23 xbars (bd23 to bd21 through bot1) - beam 1
2018-11-19 19:16:59 [DEBUG   ] board 21 xbars (bd21 from bd23 through bot5) - beam 1
2018-11-19 19:16:59 [DEBUG   ] board 21 xbars (bd21 to bd27 through bot1) - beam 1
2018-11-19 19:16:59 [DEBUG   ] board 27 xbars (bd27 from bd21 through bot5) - beam 1
2018-11-19 19:16:59 [DEBUG   ] board 27 xbars (bd27 to bd19 through bot1) - beam 1
2018-11-19 19:16:59 [DEBUG   ] board 19 xbars (bd19 from bd27 through bot5) - beam 1
2018-11-19 19:16:59 [DEBUG   ] board 19 xbars (bd19 to bd25 through bot1) - beam 1
2018-11-19 19:16:59 [DEBUG   ] board 25 xbars (bd25 from bd19 through bot5) - beam 1
2018-11-19 19:16:59 [DEBUG   ] board 25 xbars (bd25 to bd15 through bot1) - beam 1
2018-11-19 19:17:00 [DEBUG   ] board 15 xbars (board 15 from board 25 through bot5) - beam 1
2018-11-19 19:17:00 [INFO    ] Setting Beam 2 xbars
2018-11-19 19:17:00 [DEBUG   ] board 3 xbars (bd3 to bd7 through top1) - beam 2
2018-11-19 19:17:00 [DEBUG   ] board 7 xbars (bd7 from bd3 through top5) - beam 2
2018-11-19 19:17:00 [DEBUG   ] board 7 xbars (bd7 to bd11 through top1) - beam 2
2018-11-19 19:17:00 [DEBUG   ] board 11 xbars (bd11 from bd7 through top5) - beam 2
2018-11-19 19:17:00 [DEBUG   ] board 11 xbars (bd11 to bd13 through top1) - beam 2
2018-11-19 19:17:00 [DEBUG   ] board 13 xbars (bd13 from bd11 through top5) - beam 2
2018-11-19 19:17:00 [DEBUG   ] board 13 xbars (bd13 to bd9 through top1) - beam 2
2018-11-19 19:17:00 [DEBUG   ] board 9 xbars (bd9 from bd13 through top5) - beam 2
2018-11-19 19:17:00 [DEBUG   ] board 9 xbars (bd9 to bd5 through top1) - beam 2
2018-11-19 19:17:01 [DEBUG   ] board 5 xbars (bd5 from bd9 through top5) - beam 2
2018-11-19 19:17:01 [DEBUG   ] board 5 xbars (bd5 to bd2 through top1) - beam 2
2018-11-19 19:17:01 [DEBUG   ] board 2 xbars (bd2 from bd5 through top5) - beam 2
2018-11-19 19:17:01 [DEBUG   ] board 2 xbars (bd2 to bd4 through top1) - beam 2
2018-11-19 19:17:01 [DEBUG   ] board 4 xbars (bd4 from bd2 through top5) - beam 2
2018-11-19 19:17:01 [DEBUG   ] board 4 xbars (bd4 to bd6 through top1) - beam 2
2018-11-19 19:17:01 [DEBUG   ] board 6 xbars (bd6 from bd4 through top5) - beam 2
2018-11-19 19:17:01 [DEBUG   ] board 6 xbars (bd6 to bd8 through top1) - beam 2
2018-11-19 19:17:02 [DEBUG   ] board 8 xbars (bd8 from bd6 through top5) - beam 2
2018-11-19 19:17:02 [DEBUG   ] board 8 xbars (bd8 to bd10 through top1) - beam 2
2018-11-19 19:17:02 [DEBUG   ] board 10 xbars (bd10 from bd8 through top5) - beam 2
2018-11-19 19:17:02 [DEBUG   ] board 10 xbars (bd10 to bd12 through top1) - beam 2
2018-11-19 19:17:02 [DEBUG   ] board 12 xbars (bd12 from bd10 through top5) - beam 2
2018-11-19 19:17:02 [DEBUG   ] board 12 xbars (bd12 to bd14 through top1) - beam 2
2018-11-19 19:17:02 [DEBUG   ] board 14 xbars (bd14 from bd12 through bot5) - beam 2
2018-11-19 19:17:02 [DEBUG   ] board 28 xbars (bd28 to bd26 through bot5) - beam 2
2018-11-19 19:17:02 [DEBUG   ] board 26 xbars (bd26 from bd28 through top1) - beam 2
2018-11-19 19:17:03 [DEBUG   ] board 26 xbars (bd26 to bd24 through top5) - beam 2
2018-11-19 19:17:03 [DEBUG   ] board 24 xbars (bd24 from bd26 through top1) - beam 2
2018-11-19 19:17:03 [DEBUG   ] board 24 xbars (bd24 to bd22 through top5) - beam 2
2018-11-19 19:17:03 [DEBUG   ] board 22 xbars (bd22 from bd24 through top1) - beam 2
2018-11-19 19:17:03 [DEBUG   ] board 22 xbars (bd22 to bd20 through top5) - beam 2
2018-11-19 19:17:03 [DEBUG   ] board 20 xbars (bd20 from bd22 through top1) - beam 2
2018-11-19 19:17:03 [DEBUG   ] board 20 xbars (bd20 to bd18 through top5) - beam 2
2018-11-19 19:17:03 [DEBUG   ] board 18 xbars (bd18 from bd20 through top1) - beam 2
2018-11-19 19:17:03 [DEBUG   ] board 18 xbars (bd18 to bd16 through top5) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 16 xbars (bd16 from bd18 through top1) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 16 xbars (bd16 to bd19 through top5) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 19 xbars (bd19 from bd16 through top1) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 19 xbars (bd19 to bd23 through top5) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 23 xbars (bd23 from bd19 through top1) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 23 xbars (bd23 to bd27 through top5) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 27 xbars (bd27 from bd23 through top1) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 27 xbars (bd27 to bd25 through top5) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 25 xbars (bd25 from bd27 through top1) - beam 2
2018-11-19 19:17:04 [DEBUG   ] board 25 xbars (bd25 to bd21 through top5) - beam 2
2018-11-19 19:17:05 [DEBUG   ] board 21 xbars (bd21 from bd25 through top1) - beam 2
2018-11-19 19:17:05 [DEBUG   ] board 21 xbars (bd21 to bd17 through top5) - beam 2
2018-11-19 19:17:05 [DEBUG   ] board 17 xbars (bd17 from bd21 through top1) - beam 2
2018-11-19 19:17:05 [DEBUG   ] board 17 xbars (bd17 to bd15 through top5) - beam 2
2018-11-19 19:17:05 [DEBUG   ] board 15 xbars (bd15 from bd17 through top1) - beam 2
2018-11-19 19:17:05 [INFO    ] Setting Beam 3 xbars
2018-11-19 19:17:05 [DEBUG   ] board 25 xbars (bd25 to bd19 through bot5) - beam 3
2018-11-19 19:17:05 [DEBUG   ] board 19 xbars (bd19 from bd25 through bot1) - beam 3
2018-11-19 19:17:05 [DEBUG   ] board 19 xbars (bd19 to bd27 through bot5) - beam 3
2018-11-19 19:17:05 [DEBUG   ] board 27 xbars (bd27 from bd19 through bot1) - beam 3
2018-11-19 19:17:06 [DEBUG   ] board 27 xbars (bd27 to bd21 through bot5) - beam 3
2018-11-19 19:17:06 [DEBUG   ] board 21 xbars (bd21 from bd27 through bot1) - beam 3
2018-11-19 19:17:06 [DEBUG   ] board 21 xbars (bd21 to bd23 through bot5) - beam 3
2018-11-19 19:17:06 [DEBUG   ] board 23 xbars (bd23 from bd21 through bot1) - beam 3
2018-11-19 19:17:06 [DEBUG   ] board 23 xbars (bd23 to bd17 through bot5) - beam 3
2018-11-19 19:17:06 [DEBUG   ] board 17 xbars (bd17 from bd23 through bot1) - beam 3
2018-11-19 19:17:06 [DEBUG   ] board 17 xbars (bd17 to bd16 through bot5) - beam 3
2018-11-19 19:17:06 [DEBUG   ] board 16 xbars (bd16 from bd17 through bot1) - beam 3
2018-11-19 19:17:06 [DEBUG   ] board 16 xbars (bd16 to bd20 through bot5) - beam 3
2018-11-19 19:17:07 [DEBUG   ] board 20 xbars (bd20 from bd16 through bot1) - beam 3
2018-11-19 19:17:07 [DEBUG   ] board 20 xbars (bd20 to bd26 through bot5) - beam 3
2018-11-19 19:17:07 [DEBUG   ] board 26 xbars (bd26 from bd20 through bot1) - beam 3
2018-11-19 19:17:07 [DEBUG   ] board 26 xbars (bd26 to bd22 through bot5) - beam 3
2018-11-19 19:17:07 [DEBUG   ] board 22 xbars (bd22 from bd26 through bot1) - beam 3
2018-11-19 19:17:07 [DEBUG   ] board 22 xbars (bd22 to bd18 through bot5) - beam 3
2018-11-19 19:17:07 [DEBUG   ] board 18 xbars (bd18 from bd22 through bot1) - beam 3
2018-11-19 19:17:07 [DEBUG   ] board 18 xbars (bd18 to bd24 through bot5) - beam 3
2018-11-19 19:17:07 [DEBUG   ] board 24 xbars (bd24 from bd18 through bot1) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 24 xbars (bd24 to bd28 through bot5) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 28 xbars (bd28 from bd24 through top1) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 14 xbars (bd14 to bd10 through top1) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 10 xbars (bd10 from bd14 through bot5) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 10 xbars (bd10 to bd4 through bot1) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 4 xbars (bd4 from bd10 through bot5) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 4 xbars (bd4 to bd8 through bot1) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 8 xbars (bd8 from bd4 through bot5) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 8 xbars (bd8 to bd12 through bot1) - beam 3
2018-11-19 19:17:08 [DEBUG   ] board 12 xbars (bd12 from bd8 through bot5) - beam 3
2018-11-19 19:17:09 [DEBUG   ] board 12 xbars (bd12 to bd6 through bot1) - beam 3
2018-11-19 19:17:09 [DEBUG   ] board 6 xbars (bd6 from bd12 through bot5) - beam 3
2018-11-19 19:17:09 [DEBUG   ] board 6 xbars (bd6 to bd2 through bot1) - beam 3
2018-11-19 19:17:09 [DEBUG   ] board 2 xbars (bd2 from bd6 through bot5) - beam 3
2018-11-19 19:17:09 [DEBUG   ] board 2 xbars (bd2 to bd3 through bot1) - beam 3
2018-11-19 19:17:09 [DEBUG   ] board 3 xbars (bd3 from bd2 through bot5) - beam 3
2018-11-19 19:17:09 [DEBUG   ] board 3 xbars (bd3 to bd9 through bot1) - beam 3
2018-11-19 19:17:09 [DEBUG   ] board 9 xbars (bd9 from bd3 through bot5) - beam 3
2018-11-19 19:17:09 [DEBUG   ] board 9 xbars (bd9 to bd7 through bot1) - beam 3
2018-11-19 19:17:10 [DEBUG   ] board 7 xbars (bd7 from bd9 through bot5) - beam 3
2018-11-19 19:17:10 [DEBUG   ] board 7 xbars (bd7 to bd13 through bot1) - beam 3
2018-11-19 19:17:10 [DEBUG   ] board 13 xbars (bd13 from bd7 through bot5) - beam 3
2018-11-19 19:17:10 [DEBUG   ] board 13 xbars (bd13 to bd5 through bot1) - beam 3
2018-11-19 19:17:10 [DEBUG   ] board 5 xbars (bd5 from bd13 through bot5) - beam 3
2018-11-19 19:17:10 [DEBUG   ] board 5 xbars (bd5 to bd11 through bot1) - beam 3
2018-11-19 19:17:10 [DEBUG   ] board 11 xbars (bd11 from bd5 through bot5) - beam 3
2018-11-19 19:17:10 [DEBUG   ] board 11 xbars (bd11 to bd1 through bot1) - beam 3
2018-11-19 19:17:10 [DEBUG   ] board 1 xbars (bd1 from bd11 through bot5) - beam 3
2018-11-19 19:17:11 [INFO    ] Setting Beam 4 xbars
2018-11-19 19:17:11 [DEBUG   ] board 17 xbars (bd17 to bd21 through top1) - beam 4
2018-11-19 19:17:11 [DEBUG   ] board 21 xbars (bd21 from bd17 through top5) - beam 4
2018-11-19 19:17:11 [DEBUG   ] board 21 xbars (bd21 to bd25 through top1) - beam 4
2018-11-19 19:17:11 [DEBUG   ] board 21 xbars (bd21 from bd17 through top5) - beam 4
2018-11-19 19:17:11 [DEBUG   ] board 21 xbars (bd21 to bd25 through top1) - beam 4
2018-11-19 19:17:11 [DEBUG   ] board 25 xbars (bd25 from bd21 through top5) - beam 4
2018-11-19 19:17:11 [DEBUG   ] board 25 xbars (bd25 to bd27 through top1) - beam 4
2018-11-19 19:17:11 [DEBUG   ] board 27 xbars (bd27 from bd25 through top5) - beam 4
2018-11-19 19:17:11 [DEBUG   ] board 27 xbars (bd27 to bd23 through top1) - beam 4
2018-11-19 19:17:11 [DEBUG   ] board 23 xbars (bd23 from bd27 through top5) - beam 4
2018-11-19 19:17:12 [DEBUG   ] board 23 xbars (bd23 to bd19 through top1) - beam 4
2018-11-19 19:17:12 [DEBUG   ] board 19 xbars (bd19 from bd23 through top5) - beam 4
2018-11-19 19:17:12 [DEBUG   ] board 19 xbars (bd19 to bd16 through top1) - beam 4
2018-11-19 19:17:12 [DEBUG   ] board 16 xbars (bd16 from bd19 through top5) - beam 4
2018-11-19 19:17:12 [DEBUG   ] board 16 xbars (bd16 to bd18 through top1) - beam 4
2018-11-19 19:17:12 [DEBUG   ] board 18 xbars (bd18 from bd16 through top5) - beam 4
2018-11-19 19:17:12 [DEBUG   ] board 18 xbars (bd18 to bd20 through top1) - beam 4
2018-11-19 19:17:12 [DEBUG   ] board 20 xbars (bd20 from bd18 through top5) - beam 4
2018-11-19 19:17:12 [DEBUG   ] board 20 xbars (bd20 to bd22 through top1) - beam 4
2018-11-19 19:17:13 [DEBUG   ] board 22 xbars (bd22 from bd20 through top5) - beam 4
2018-11-19 19:17:13 [DEBUG   ] board 22 xbars (bd22 to bd24 through top1) - beam 4
2018-11-19 19:17:13 [DEBUG   ] board 24 xbars (bd24 from bd22 through top5) - beam 4
2018-11-19 19:17:13 [DEBUG   ] board 24 xbars (bd24 to bd26 through top1) - beam 4
2018-11-19 19:17:13 [DEBUG   ] board 26 xbars (bd26 from bd24 through top5) - beam 4
2018-11-19 19:17:13 [DEBUG   ] board 26 xbars (bd26 to bd28 through top1) - beam 4
2018-11-19 19:17:13 [DEBUG   ] board 28 xbars (bd28 from bd26 through bot5) - beam 4
2018-11-19 19:17:13 [DEBUG   ] board 14 xbars (bd14 to bd12 through bot5) - beam 4
2018-11-19 19:17:13 [DEBUG   ] board 12 xbars (bd12 from bd14 through top1) - beam 4
2018-11-19 19:17:14 [DEBUG   ] board 12 xbars (bd12 to bd10 through top5) - beam 4
2018-11-19 19:17:14 [DEBUG   ] board 10 xbars (bd10 from bd12 through top1) - beam 4
2018-11-19 19:17:14 [DEBUG   ] board 10 xbars (bd10 to bd8 through top5) - beam 4
2018-11-19 19:17:14 [DEBUG   ] board 8 xbars (bd8 from bd10 through top1) - beam 4
2018-11-19 19:17:14 [DEBUG   ] board 8 xbars (bd8 to bd6 through top5) - beam 4
2018-11-19 19:17:14 [DEBUG   ] board 6 xbars (bd6 from bd8 through top1) - beam 4
2018-11-19 19:17:14 [DEBUG   ] board 6 xbars (bd6 to bd4 through top5) - beam 4
2018-11-19 19:17:14 [DEBUG   ] board 4 xbars (bd4 from bd6 through top1) - beam 4
2018-11-19 19:17:14 [DEBUG   ] board 4 xbars (bd4 to bd2 through top5) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 2 xbars (bd2 from bd4 through top1) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 2 xbars (bd2 to bd5 through top5) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 5 xbars (bd5 from bd2 through top1) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 5 xbars (bd5 to bd9 through top5) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 9 xbars (bd9 from bd5 through top1) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 9 xbars (bd9 to bd13 through top5) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 13 xbars (bd13 from bd9 through top1) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 13 xbars (bd13 to bd11 through top5) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 11 xbars (bd11 from bd13 through top1) - beam 4
2018-11-19 19:17:15 [DEBUG   ] board 11 xbars (bd11 to bd7 through top5) - beam 4
2018-11-19 19:17:16 [DEBUG   ] board 7 xbars (bd7 from bd11 through top1) - beam 4
2018-11-19 19:17:16 [DEBUG   ] board 7 xbars (bd7 to bd3 through top5) - beam 4
2018-11-19 19:17:16 [DEBUG   ] board 3 xbars (bd3 from bd7 through top1) - beam 4
2018-11-19 19:17:16 [DEBUG   ] board 3 xbars (bd3 to bd1 through top5) - beam 4
2018-11-19 19:17:16 [DEBUG   ] board 1 xbars (bd1 from bd3 through top1) - beam 4
2018-11-19 19:17:16 [INFO    ] All xbars set
2018-11-19 19:17:16 [INFO    ] Setting adder chain routing for all beams
2018-11-19 19:17:21 [INFO    ] Setting beam ids
2018-11-19 19:17:31 [INFO    ] Starting beam delay calibration
2018-11-19 19:17:43 [INFO    ] Beam 1 latency is 5722
2018-11-19 19:17:43 [INFO    ] Beam 2 latency is 5760
2018-11-19 19:17:43 [INFO    ] Beam 3 latency is 5714
2018-11-19 19:17:43 [INFO    ] Beam 4 latency is 5724
2018-11-19 19:17:43 [INFO    ] Stop calibration
2018-11-19 19:17:51 [INFO    ] Beam delay calibration successful
2018-11-19 19:17:51 [INFO    ] Calibration passed
2018-11-19 19:17:51 [DEBUG   ] spawn_process: at 2018-11-19 19:17:51 running ['/usr/local/bin/lwa34', '/lwa/software/cfg_eth 4 1 192.168.0.1 192.168.0.33 10000']
2018-11-19 19:17:51 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 5 0x138 8909
2018-11-19 19:17:51 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 5 0x13c 9111
2018-11-19 19:17:51 [DEBUG   ] spawn_process: at 2018-11-19 19:17:51 running ['/usr/local/bin/lwa34', '/lwa/software/cfg_eth 2 1 192.168.0.2 192.168.0.34 10000']
2018-11-19 19:17:51 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 1 0x138 8a0a
2018-11-19 19:17:51 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 1 0x13c 9212
2018-11-19 19:17:51 [DEBUG   ] spawn_process: at 2018-11-19 19:17:51 running ['/usr/local/bin/lwa16', '/lwa/software/cfg_eth 4 1 192.168.0.3 192.168.0.35 10000']
2018-11-19 19:17:51 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 5 0x138 8b0b
2018-11-19 19:17:51 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 5 0x13c 9313
2018-11-19 19:17:51 [DEBUG   ] spawn_process: at 2018-11-19 19:17:51 running ['/usr/local/bin/lwa16', '/lwa/software/cfg_eth 2 1 192.168.0.4 192.168.0.36 10000']
2018-11-19 19:17:51 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 1 0x138 8c0c
2018-11-19 19:17:51 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 1 0x13c 9414
2018-11-19 19:17:51 [DEBUG   ] SetDRXTime: DP2 - DP/MCS NTP skew is -2 msec
2018-11-19 19:17:52 [DEBUG   ] SetDRXTime: started set_drx_time on hostname lwa16 at 290 ms (1542655072.292 s) with PID 4559 and time 1542655073
2018-11-19 19:17:52 [DEBUG   ] SetDRXTime: DP2 - DP/MCS NTP skew is -3 msec
2018-11-19 19:17:52 [DEBUG   ] SetDRXTime: started set_drx_time on hostname lwa34 at 379 ms (1542655072.381 s) with PID 4562 and time 1542655073
2018-11-19 19:17:53 [DEBUG   ] Finished updating ant stats in 0.866 seconds
2018-11-19 19:17:53 [INFO    ] Started antennas statistics background thread
2018-11-19 19:17:53 [INFO    ] Started board flags background thread
2018-11-19 19:17:53 [INFO    ] Started BAM background thread
2018-11-19 19:17:53 [INFO    ] Finished the INI process (ref. #561334) in 108.020 s
2018-11-19 19:17:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:17:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 19:17:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:17:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:17:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:17:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:17:54 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:17:54 [DEBUG   ] Finished updating board flags in 1.053 seconds
2018-11-19 19:18:13 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:18:33 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:18:53 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:18:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 19:18:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 19:18:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:18:54 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 19:18:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 19:18:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:18:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:18:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:18:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:18:54 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:18:54 [DEBUG   ] Finished updating board flags in 0.990 seconds
2018-11-19 19:19:05 [DEBUG   ] Got command RPT from MCS: ref #561339
2018-11-19 19:19:05 [DEBUG   ] summary = ERROR
2018-11-19 19:19:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561339  13 58441 69545493 A  ERRORERROR'
2018-11-19 19:19:05 [DEBUG   ] Got command RPT from MCS: ref #561340
2018-11-19 19:19:05 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:19:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561340  77 58441 69545494 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:19:13 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:19:33 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:19:53 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:19:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 19:19:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:19:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:19:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:19:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:19:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:19:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:19:55 [DEBUG   ] Finished updating board flags in 1.849 seconds
2018-11-19 19:19:58 [DEBUG   ] Got command RPT from MCS: ref #561343
2018-11-19 19:19:58 [DEBUG   ] summary = ERROR
2018-11-19 19:19:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561343  13 58441 69598485 A  ERRORERROR'
2018-11-19 19:19:58 [DEBUG   ] Got command RPT from MCS: ref #561344
2018-11-19 19:19:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:19:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561344  77 58441 69598689 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:19:58 [DEBUG   ] Got command RPT from MCS: ref #561345
2018-11-19 19:19:58 [DEBUG   ] summary = ERROR
2018-11-19 19:19:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561345  13 58441 69598894 A  ERRORERROR'
2018-11-19 19:19:59 [DEBUG   ] Got command RPT from MCS: ref #561347
2018-11-19 19:19:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:19:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561347  77 58441 69599097 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:19:59 [DEBUG   ] Got command RPT from MCS: ref #561348
2018-11-19 19:19:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:19:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561348  12 58441 69599289 A  ERRORB™™š'
2018-11-19 19:20:13 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:20:33 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:20:46 [DEBUG   ] Got command RPT from MCS: ref #561353
2018-11-19 19:20:46 [DEBUG   ] summary = ERROR
2018-11-19 19:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561353  13 58441 69646477 A  ERRORERROR'
2018-11-19 19:20:46 [DEBUG   ] Got command RPT from MCS: ref #561354
2018-11-19 19:20:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561354  77 58441 69646478 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:20:53 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:20:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:20:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:20:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:20:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:20:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:20:54 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:20:54 [DEBUG   ] Finished updating board flags in 1.217 seconds
2018-11-19 19:21:13 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:21:33 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:21:53 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:21:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:21:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:21:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 19:21:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:21:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:21:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:21:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:21:54 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:21:54 [DEBUG   ] Finished updating board flags in 1.247 seconds
2018-11-19 19:22:13 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:22:33 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:22:53 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:22:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:22:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 19:22:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:22:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:22:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:22:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:22:54 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:22:54 [DEBUG   ] Finished updating board flags in 1.215 seconds
2018-11-19 19:23:13 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:23:33 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:23:53 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:23:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:23:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:23:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:23:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:23:54 [WARNING ] (13, 1, 'bot')
2018-11-19 19:23:54 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:23:54 [DEBUG   ] Finished updating board flags in 1.118 seconds
2018-11-19 19:24:13 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:24:33 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:24:53 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:24:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:24:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:24:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:24:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:24:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:24:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:24:55 [DEBUG   ] Finished updating board flags in 1.598 seconds
2018-11-19 19:24:58 [DEBUG   ] Got command RPT from MCS: ref #561374
2018-11-19 19:24:58 [DEBUG   ] summary = ERROR
2018-11-19 19:24:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561374  13 58441 69898569 A  ERRORERROR'
2018-11-19 19:24:58 [DEBUG   ] Got command RPT from MCS: ref #561375
2018-11-19 19:24:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:24:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561375  77 58441 69898773 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:24:58 [DEBUG   ] Got command RPT from MCS: ref #561376
2018-11-19 19:24:58 [DEBUG   ] summary = ERROR
2018-11-19 19:24:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561376  13 58441 69898977 A  ERRORERROR'
2018-11-19 19:24:59 [DEBUG   ] Got command RPT from MCS: ref #561377
2018-11-19 19:24:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:24:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561377  77 58441 69899181 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:24:59 [DEBUG   ] Got command RPT from MCS: ref #561378
2018-11-19 19:24:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:24:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561378  12 58441 69899373 A  ERRORB™™š'
2018-11-19 19:25:13 [DEBUG   ] Finished updating ant stats in 0.795 seconds
2018-11-19 19:25:33 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:25:46 [DEBUG   ] Got command RPT from MCS: ref #561383
2018-11-19 19:25:46 [DEBUG   ] summary = ERROR
2018-11-19 19:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561383  13 58441 69946485 A  ERRORERROR'
2018-11-19 19:25:46 [DEBUG   ] Got command RPT from MCS: ref #561384
2018-11-19 19:25:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561384  77 58441 69946486 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:25:53 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:25:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:25:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:25:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:25:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:25:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:25:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:25:55 [DEBUG   ] Finished updating board flags in 1.366 seconds
2018-11-19 19:26:13 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:26:33 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 19:26:53 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:26:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:26:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 19:26:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 19:26:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:26:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:26:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:26:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:26:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:26:55 [DEBUG   ] Finished updating board flags in 1.234 seconds
2018-11-19 19:27:13 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:27:33 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:27:53 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 19:27:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:27:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:27:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:27:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:27:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:27:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:27:55 [DEBUG   ] Finished updating board flags in 1.176 seconds
2018-11-19 19:28:13 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 19:28:33 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 19:28:53 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 19:28:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:28:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:28:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:28:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:28:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:28:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:28:55 [DEBUG   ] Finished updating board flags in 1.021 seconds
2018-11-19 19:29:13 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:29:33 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 19:29:53 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 19:29:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:29:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:29:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:29:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:29:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:29:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:29:55 [DEBUG   ] Finished updating board flags in 1.020 seconds
2018-11-19 19:29:58 [DEBUG   ] Got command RPT from MCS: ref #561406
2018-11-19 19:29:58 [DEBUG   ] summary = ERROR
2018-11-19 19:29:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561406  13 58441 70198657 A  ERRORERROR'
2018-11-19 19:29:58 [DEBUG   ] Got command RPT from MCS: ref #561407
2018-11-19 19:29:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:29:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561407  77 58441 70198861 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:29:59 [DEBUG   ] Got command RPT from MCS: ref #561408
2018-11-19 19:29:59 [DEBUG   ] summary = ERROR
2018-11-19 19:29:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561408  13 58441 70199065 A  ERRORERROR'
2018-11-19 19:29:59 [DEBUG   ] Got command RPT from MCS: ref #561409
2018-11-19 19:29:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:29:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561409  77 58441 70199257 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:29:59 [DEBUG   ] Got command RPT from MCS: ref #561411
2018-11-19 19:29:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:29:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561411  12 58441 70199461 A  ERRORB™™š'
2018-11-19 19:30:13 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 19:30:33 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:30:46 [DEBUG   ] Got command RPT from MCS: ref #561416
2018-11-19 19:30:46 [DEBUG   ] summary = ERROR
2018-11-19 19:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561416  13 58441 70246485 A  ERRORERROR'
2018-11-19 19:30:46 [DEBUG   ] Got command RPT from MCS: ref #561417
2018-11-19 19:30:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561417  77 58441 70246486 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:30:53 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:30:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:30:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:30:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:30:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:30:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:30:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:30:56 [DEBUG   ] Finished updating board flags in 1.994 seconds
2018-11-19 19:31:13 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:31:33 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:31:53 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:31:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:31:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:31:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:31:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:31:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:31:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:31:55 [DEBUG   ] Finished updating board flags in 1.249 seconds
2018-11-19 19:32:13 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:32:33 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:32:53 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:32:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:32:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:32:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:32:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:32:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:32:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:32:55 [DEBUG   ] Finished updating board flags in 1.241 seconds
2018-11-19 19:33:13 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:33:33 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:33:53 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:33:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:33:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:33:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:33:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:33:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:33:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:33:55 [DEBUG   ] Finished updating board flags in 1.114 seconds
2018-11-19 19:34:13 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:34:33 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:34:53 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:34:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:34:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:34:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:34:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:34:55 [WARNING ] (13, 1, 'bot')
2018-11-19 19:34:55 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:34:55 [DEBUG   ] Finished updating board flags in 1.041 seconds
2018-11-19 19:34:58 [DEBUG   ] Got command RPT from MCS: ref #561441
2018-11-19 19:34:58 [DEBUG   ] summary = ERROR
2018-11-19 19:34:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561441  13 58441 70498749 A  ERRORERROR'
2018-11-19 19:34:58 [DEBUG   ] Got command RPT from MCS: ref #561442
2018-11-19 19:34:58 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:34:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561442  77 58441 70498941 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:34:59 [DEBUG   ] Got command RPT from MCS: ref #561443
2018-11-19 19:34:59 [DEBUG   ] summary = ERROR
2018-11-19 19:34:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561443  13 58441 70499145 A  ERRORERROR'
2018-11-19 19:34:59 [DEBUG   ] Got command RPT from MCS: ref #561444
2018-11-19 19:34:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:34:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561444  77 58441 70499349 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:34:59 [DEBUG   ] Got command RPT from MCS: ref #561445
2018-11-19 19:34:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:34:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561445  12 58441 70499553 A  ERRORB˜™š'
2018-11-19 19:35:13 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:35:33 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:35:46 [DEBUG   ] Got command RPT from MCS: ref #561446
2018-11-19 19:35:46 [DEBUG   ] summary = ERROR
2018-11-19 19:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561446  13 58441 70546481 A  ERRORERROR'
2018-11-19 19:35:46 [DEBUG   ] Got command RPT from MCS: ref #561447
2018-11-19 19:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561447  77 58441 70546482 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:35:53 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:35:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 19:35:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:35:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:35:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:35:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:35:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:35:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:35:56 [DEBUG   ] Finished updating board flags in 1.562 seconds
2018-11-19 19:36:13 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:36:20 [DEBUG   ] Got command RPT from MCS: ref #561463
2018-11-19 19:36:20 [DEBUG   ] summary = ERROR
2018-11-19 19:36:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561463  13 58441 70580434 A  ERRORERROR'
2018-11-19 19:36:20 [DEBUG   ] Got command RPT from MCS: ref #561464
2018-11-19 19:36:20 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:36:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561464  77 58441 70580625 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:36:20 [DEBUG   ] Got command RPT from MCS: ref #561465
2018-11-19 19:36:20 [DEBUG   ] tbn freq = 0.0
2018-11-19 19:36:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561465  12 58441 70580829 A  ERROR'
2018-11-19 19:36:21 [DEBUG   ] Got command STP from MCS: ref #561466
2018-11-19 19:36:21 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   561466  77 58441 70581033 R  ERROR0x0B! STP: Other error running command - TBN is not currently running'
2018-11-19 19:36:32 [DEBUG   ] Got command BAM from MCS: ref #561475
2018-11-19 19:36:32 [INFO    ] Finished processing BAM request
2018-11-19 19:36:32 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:36:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561475   8 58441 70592029 A  ERROR'
2018-11-19 19:36:32 [DEBUG   ] Got command BAM from MCS: ref #561476
2018-11-19 19:36:32 [INFO    ] Finished processing BAM request
2018-11-19 19:36:32 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:36:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561476   8 58441 70592033 A  ERROR'
2018-11-19 19:36:32 [DEBUG   ] Got command DRX from MCS: ref #561477
2018-11-19 19:36:32 [INFO    ] sending drx command to lwa16
2018-11-19 19:36:32 [DEBUG   ] run_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa16', '/lwa/software/drx 1 1 35100000.0 7 9 9']
2018-11-19 19:36:32 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:36:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561477   8 58441 70592097 A  ERROR'
2018-11-19 19:36:32 [DEBUG   ] Got command DRX from MCS: ref #561478
2018-11-19 19:36:32 [INFO    ] sending drx command to lwa34
2018-11-19 19:36:32 [DEBUG   ] run_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa34', '/lwa/software/drx 1 1 64500000.0 7 9 9']
2018-11-19 19:36:32 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:36:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561478   8 58441 70592101 A  ERROR'
2018-11-19 19:36:32 [DEBUG   ] Got command DRX from MCS: ref #561479
2018-11-19 19:36:32 [INFO    ] sending drx command to lwa16
2018-11-19 19:36:32 [DEBUG   ] run_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa16', '/lwa/software/drx 1 2 49800000.0 7 9 9']
2018-11-19 19:36:32 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:36:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561479   8 58441 70592105 A  ERROR'
2018-11-19 19:36:32 [DEBUG   ] Got command DRX from MCS: ref #561480
2018-11-19 19:36:32 [INFO    ] sending drx command to lwa34
2018-11-19 19:36:32 [DEBUG   ] run_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa34', '/lwa/software/drx 1 2 79200000.0 7 9 9']
2018-11-19 19:36:32 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:36:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561480   8 58441 70592109 A  ERROR'
2018-11-19 19:36:32 [INFO    ] Finished starting DRX
2018-11-19 19:36:32 [INFO    ] Finished starting DRX
2018-11-19 19:36:32 [INFO    ] Finished starting DRX
2018-11-19 19:36:32 [INFO    ] Finished starting DRX
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 19:36:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 19:36:32 [DEBUG   ] spawn_process: at 2018-11-19 19:36:32 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 19:36:33 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:36:50 [DEBUG   ] Got command DRX from MCS: ref #561485
2018-11-19 19:36:50 [INFO    ] sending drx command to lwa16
2018-11-19 19:36:50 [DEBUG   ] run_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 1 78000000.0 6 9 0']
2018-11-19 19:36:50 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:36:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561485   8 58441 70610009 A  ERROR'
2018-11-19 19:36:50 [DEBUG   ] Got command DRX from MCS: ref #561486
2018-11-19 19:36:50 [INFO    ] sending drx command to lwa16
2018-11-19 19:36:50 [DEBUG   ] run_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 2 76000000.0 6 9 0']
2018-11-19 19:36:50 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:36:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561486   8 58441 70610013 A  ERROR'
2018-11-19 19:36:50 [DEBUG   ] Got command BAM from MCS: ref #561487
2018-11-19 19:36:50 [INFO    ] Finished processing BAM request
2018-11-19 19:36:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:36:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561487   8 58441 70610029 A  ERROR'
2018-11-19 19:36:50 [INFO    ] Finished starting DRX
2018-11-19 19:36:50 [INFO    ] Finished starting DRX
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 19:36:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 19:36:50 [DEBUG   ] spawn_process: at 2018-11-19 19:36:50 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 19:36:53 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:36:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 19:36:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 19:36:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 19:36:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 19:36:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:36:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:36:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:36:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:36:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:36:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:36:56 [DEBUG   ] Finished updating board flags in 1.368 seconds
2018-11-19 19:37:13 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:37:14 [DEBUG   ] Got command BAM from MCS: ref #561493
2018-11-19 19:37:14 [INFO    ] Finished processing BAM request
2018-11-19 19:37:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:37:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561493   8 58441 70634033 A  ERROR'
2018-11-19 19:37:14 [DEBUG   ] Got command BAM from MCS: ref #561494
2018-11-19 19:37:14 [INFO    ] Finished processing BAM request
2018-11-19 19:37:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:37:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561494   8 58441 70634037 A  ERROR'
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 19:37:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 19:37:14 [DEBUG   ] spawn_process: at 2018-11-19 19:37:14 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 19:37:21 [DEBUG   ] Got command STP from MCS: ref #561495
2018-11-19 19:37:21 [INFO    ] Finished processing BAM request
2018-11-19 19:37:21 [INFO    ] Shutdown the adder chain on beam #3
2018-11-19 19:37:21 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   561495   8 58441 70641025 A  ERROR'
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 19:37:21 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 19:37:21 [DEBUG   ] spawn_process: at 2018-11-19 19:37:21 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 19:37:33 [DEBUG   ] Finished updating ant stats in 0.399 seconds
2018-11-19 19:37:50 [DEBUG   ] Got command RPT from MCS: ref #561496
2018-11-19 19:37:50 [DEBUG   ] summary = ERROR
2018-11-19 19:37:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561496  13 58441 70670269 A  ERRORERROR'
2018-11-19 19:37:50 [DEBUG   ] Got command RPT from MCS: ref #561497
2018-11-19 19:37:50 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:37:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561497  77 58441 70670461 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:37:50 [DEBUG   ] Got command RPT from MCS: ref #561498
2018-11-19 19:37:50 [DEBUG   ] tbn freq = 0.0
2018-11-19 19:37:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561498  12 58441 70670665 A  ERROR'
2018-11-19 19:37:50 [DEBUG   ] Got command STP from MCS: ref #561499
2018-11-19 19:37:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   561499  77 58441 70670869 R  ERROR0x0B! STP: Other error running command - TBN is not currently running'
2018-11-19 19:37:54 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 19:37:55 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:37:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 19:37:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:37:56 [DEBUG   ] Got command BAM from MCS: ref #561505
2018-11-19 19:37:56 [INFO    ] Finished processing BAM request
2018-11-19 19:37:56 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:37:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561505   8 58441 70676025 A  ERROR'
2018-11-19 19:37:56 [DEBUG   ] Got command BAM from MCS: ref #561504
2018-11-19 19:37:56 [INFO    ] Finished processing BAM request
2018-11-19 19:37:56 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:37:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561504   8 58441 70676029 A  ERROR'
2018-11-19 19:37:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 19:37:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:37:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:37:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:37:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:37:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:37:56 [DEBUG   ] Finished updating board flags in 1.219 seconds
2018-11-19 19:37:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 19:37:56 [DEBUG   ] spawn_process: at 2018-11-19 19:37:56 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 19:37:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 19:37:57 [DEBUG   ] spawn_process: at 2018-11-19 19:37:57 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 19:38:14 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 19:38:19 [DEBUG   ] Got command DRX from MCS: ref #561508
2018-11-19 19:38:19 [INFO    ] sending drx command to lwa16
2018-11-19 19:38:19 [DEBUG   ] run_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 1 78000000.0 6 9 0']
2018-11-19 19:38:19 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:38:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561508   8 58441 70699009 A  ERROR'
2018-11-19 19:38:19 [DEBUG   ] Got command DRX from MCS: ref #561509
2018-11-19 19:38:19 [INFO    ] sending drx command to lwa16
2018-11-19 19:38:19 [DEBUG   ] run_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 2 76000000.0 6 9 0']
2018-11-19 19:38:19 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:38:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561509   8 58441 70699013 A  ERROR'
2018-11-19 19:38:19 [DEBUG   ] Got command BAM from MCS: ref #561510
2018-11-19 19:38:19 [INFO    ] Finished processing BAM request
2018-11-19 19:38:19 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:38:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561510   8 58441 70699029 A  ERROR'
2018-11-19 19:38:19 [INFO    ] Finished starting DRX
2018-11-19 19:38:19 [INFO    ] Finished starting DRX
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 19:38:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 19:38:19 [DEBUG   ] spawn_process: at 2018-11-19 19:38:19 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 19:38:34 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 19:38:38 [DEBUG   ] Got command BAM from MCS: ref #561513
2018-11-19 19:38:38 [INFO    ] Finished processing BAM request
2018-11-19 19:38:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:38:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561513   8 58441 70718029 A  ERROR'
2018-11-19 19:38:38 [DEBUG   ] Got command BAM from MCS: ref #561512
2018-11-19 19:38:38 [INFO    ] Finished processing BAM request
2018-11-19 19:38:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:38:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561512   8 58441 70718033 A  ERROR'
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 19:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 19:38:38 [DEBUG   ] spawn_process: at 2018-11-19 19:38:38 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 19:38:50 [DEBUG   ] Got command STP from MCS: ref #561514
2018-11-19 19:38:50 [INFO    ] Finished processing BAM request
2018-11-19 19:38:50 [INFO    ] Shutdown the adder chain on beam #3
2018-11-19 19:38:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   561514   8 58441 70730025 A  ERROR'
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 19:38:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 19:38:50 [DEBUG   ] spawn_process: at 2018-11-19 19:38:50 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 19:38:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:38:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 19:38:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:38:56 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 19:38:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:38:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:38:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:38:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:38:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:38:56 [DEBUG   ] Finished updating board flags in 1.186 seconds
2018-11-19 19:39:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:39:20 [DEBUG   ] Got command BAM from MCS: ref #561515
2018-11-19 19:39:20 [INFO    ] Finished processing BAM request
2018-11-19 19:39:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:39:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561515   8 58441 70760033 A  ERROR'
2018-11-19 19:39:20 [DEBUG   ] Got command BAM from MCS: ref #561516
2018-11-19 19:39:20 [INFO    ] Finished processing BAM request
2018-11-19 19:39:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:39:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561516   8 58441 70760037 A  ERROR'
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:39:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:39:20 [DEBUG   ] spawn_process: at 2018-11-19 19:39:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:39:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:39:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:39:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:39:56 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 19:39:56 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 19:39:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:39:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:39:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:39:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:39:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:39:56 [DEBUG   ] Finished updating board flags in 1.042 seconds
2018-11-19 19:39:58 [DEBUG   ] Got command RPT from MCS: ref #561519
2018-11-19 19:39:58 [DEBUG   ] summary = ERROR
2018-11-19 19:39:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561519  13 58441 70798825 A  ERRORERROR'
2018-11-19 19:39:59 [DEBUG   ] Got command RPT from MCS: ref #561520
2018-11-19 19:39:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:39:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561520  77 58441 70799029 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:39:59 [DEBUG   ] Got command RPT from MCS: ref #561521
2018-11-19 19:39:59 [DEBUG   ] summary = ERROR
2018-11-19 19:39:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561521  13 58441 70799233 A  ERRORERROR'
2018-11-19 19:39:59 [DEBUG   ] Got command RPT from MCS: ref #561523
2018-11-19 19:39:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:39:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561523  77 58441 70799437 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:39:59 [DEBUG   ] Got command RPT from MCS: ref #561524
2018-11-19 19:39:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:39:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561524  12 58441 70799629 A  ERRORB¡ff'
2018-11-19 19:40:01 [DEBUG   ] Got command BAM from MCS: ref #561528
2018-11-19 19:40:01 [INFO    ] Finished processing BAM request
2018-11-19 19:40:01 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:40:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561528   8 58441 70801025 A  ERROR'
2018-11-19 19:40:01 [DEBUG   ] Got command BAM from MCS: ref #561527
2018-11-19 19:40:01 [INFO    ] Finished processing BAM request
2018-11-19 19:40:01 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:40:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561527   8 58441 70801029 A  ERROR'
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 19:40:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 19:40:01 [DEBUG   ] spawn_process: at 2018-11-19 19:40:01 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 19:40:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:40:19 [DEBUG   ] Got command RPT from MCS: ref #561535
2018-11-19 19:40:19 [DEBUG   ] summary = ERROR
2018-11-19 19:40:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561535  13 58441 70819753 A  ERRORERROR'
2018-11-19 19:40:19 [DEBUG   ] Got command RPT from MCS: ref #561536
2018-11-19 19:40:19 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:40:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561536  77 58441 70819957 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:40:20 [DEBUG   ] Got command RPT from MCS: ref #561537
2018-11-19 19:40:20 [DEBUG   ] tbn freq = 0.0
2018-11-19 19:40:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561537  12 58441 70820161 A  ERROR'
2018-11-19 19:40:20 [DEBUG   ] Got command STP from MCS: ref #561538
2018-11-19 19:40:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   561538  77 58441 70820353 R  ERROR0x0B! STP: Other error running command - TBN is not currently running'
2018-11-19 19:40:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:40:42 [DEBUG   ] Got command BAM from MCS: ref #561544
2018-11-19 19:40:42 [INFO    ] Finished processing BAM request
2018-11-19 19:40:42 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:40:42 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561544   8 58441 70842029 A  ERROR'
2018-11-19 19:40:42 [DEBUG   ] Got command BAM from MCS: ref #561543
2018-11-19 19:40:42 [INFO    ] Finished processing BAM request
2018-11-19 19:40:42 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:40:42 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561543   8 58441 70842033 A  ERROR'
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 19:40:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 19:40:42 [DEBUG   ] spawn_process: at 2018-11-19 19:40:42 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 19:40:46 [DEBUG   ] Got command RPT from MCS: ref #561547
2018-11-19 19:40:46 [DEBUG   ] summary = ERROR
2018-11-19 19:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561547  13 58441 70846485 A  ERRORERROR'
2018-11-19 19:40:46 [DEBUG   ] Got command RPT from MCS: ref #561548
2018-11-19 19:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561548  77 58441 70846486 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:40:49 [DEBUG   ] Got command DRX from MCS: ref #561564
2018-11-19 19:40:49 [INFO    ] sending drx command to lwa16
2018-11-19 19:40:49 [DEBUG   ] run_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 1 78000000.0 6 9 0']
2018-11-19 19:40:49 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:40:49 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561564   8 58441 70849009 A  ERROR'
2018-11-19 19:40:49 [DEBUG   ] Got command DRX from MCS: ref #561565
2018-11-19 19:40:49 [INFO    ] sending drx command to lwa16
2018-11-19 19:40:49 [DEBUG   ] run_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 2 76000000.0 6 9 0']
2018-11-19 19:40:49 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:40:49 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561565   8 58441 70849013 A  ERROR'
2018-11-19 19:40:49 [DEBUG   ] Got command BAM from MCS: ref #561566
2018-11-19 19:40:49 [INFO    ] Finished processing BAM request
2018-11-19 19:40:49 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:40:49 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561566   8 58441 70849033 A  ERROR'
2018-11-19 19:40:49 [INFO    ] Finished starting DRX
2018-11-19 19:40:49 [INFO    ] Finished starting DRX
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 19:40:49 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 19:40:49 [DEBUG   ] spawn_process: at 2018-11-19 19:40:49 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 19:40:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:40:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 19:40:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 19:40:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 19:40:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:40:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 19:40:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 19:40:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 19:40:57 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 19:40:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 19:40:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:40:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:40:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:40:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:40:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:40:57 [DEBUG   ] Finished updating board flags in 2.014 seconds
2018-11-19 19:41:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:41:20 [DEBUG   ] Got command STP from MCS: ref #561568
2018-11-19 19:41:20 [INFO    ] Finished processing BAM request
2018-11-19 19:41:20 [INFO    ] Shutdown the adder chain on beam #3
2018-11-19 19:41:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   561568   8 58441 70880025 A  ERROR'
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 19:41:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 19:41:20 [DEBUG   ] spawn_process: at 2018-11-19 19:41:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 19:41:23 [DEBUG   ] Got command BAM from MCS: ref #561570
2018-11-19 19:41:23 [INFO    ] Finished processing BAM request
2018-11-19 19:41:23 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:41:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561570   8 58441 70883033 A  ERROR'
2018-11-19 19:41:23 [DEBUG   ] Got command BAM from MCS: ref #561569
2018-11-19 19:41:23 [INFO    ] Finished processing BAM request
2018-11-19 19:41:23 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:41:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561569   8 58441 70883037 A  ERROR'
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 19:41:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 19:41:23 [DEBUG   ] spawn_process: at 2018-11-19 19:41:23 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 19:41:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:41:49 [DEBUG   ] Got command RPT from MCS: ref #561571
2018-11-19 19:41:49 [DEBUG   ] summary = ERROR
2018-11-19 19:41:49 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561571  13 58441 70909341 A  ERRORERROR'
2018-11-19 19:41:49 [DEBUG   ] Got command RPT from MCS: ref #561572
2018-11-19 19:41:49 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:41:49 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561572  77 58441 70909545 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:41:49 [DEBUG   ] Got command RPT from MCS: ref #561573
2018-11-19 19:41:49 [DEBUG   ] tbn freq = 0.0
2018-11-19 19:41:49 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561573  12 58441 70909749 A  ERROR'
2018-11-19 19:41:49 [DEBUG   ] Got command STP from MCS: ref #561574
2018-11-19 19:41:49 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   561574  77 58441 70909953 R  ERROR0x0B! STP: Other error running command - TBN is not currently running'
2018-11-19 19:41:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:41:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 19:41:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:41:56 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 19:41:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:41:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:41:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:41:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:41:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:41:56 [DEBUG   ] Finished updating board flags in 1.408 seconds
2018-11-19 19:42:04 [DEBUG   ] Got command BAM from MCS: ref #561580
2018-11-19 19:42:04 [INFO    ] Finished processing BAM request
2018-11-19 19:42:04 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:42:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561580   8 58441 70924033 A  ERROR'
2018-11-19 19:42:04 [DEBUG   ] Got command BAM from MCS: ref #561579
2018-11-19 19:42:04 [INFO    ] Finished processing BAM request
2018-11-19 19:42:04 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:42:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561579   8 58441 70924037 A  ERROR'
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 19:42:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 19:42:04 [DEBUG   ] spawn_process: at 2018-11-19 19:42:04 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 19:42:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:42:19 [DEBUG   ] Got command DRX from MCS: ref #561583
2018-11-19 19:42:19 [INFO    ] sending drx command to lwa16
2018-11-19 19:42:19 [DEBUG   ] run_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 1 78000000.0 6 9 0']
2018-11-19 19:42:19 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:42:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561583   8 58441 70939009 A  ERROR'
2018-11-19 19:42:19 [DEBUG   ] Got command DRX from MCS: ref #561584
2018-11-19 19:42:19 [INFO    ] sending drx command to lwa16
2018-11-19 19:42:19 [DEBUG   ] run_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 2 76000000.0 6 9 0']
2018-11-19 19:42:19 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 19:42:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   561584   8 58441 70939013 A  ERROR'
2018-11-19 19:42:19 [DEBUG   ] Got command BAM from MCS: ref #561585
2018-11-19 19:42:19 [INFO    ] Finished processing BAM request
2018-11-19 19:42:19 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:42:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561585   8 58441 70939029 A  ERROR'
2018-11-19 19:42:19 [INFO    ] Finished starting DRX
2018-11-19 19:42:19 [INFO    ] Finished starting DRX
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 19:42:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 19:42:19 [DEBUG   ] spawn_process: at 2018-11-19 19:42:19 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 19:42:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:42:45 [DEBUG   ] Got command BAM from MCS: ref #561587
2018-11-19 19:42:45 [INFO    ] Finished processing BAM request
2018-11-19 19:42:45 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:42:45 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561587   8 58441 70965025 A  ERROR'
2018-11-19 19:42:45 [DEBUG   ] Got command BAM from MCS: ref #561588
2018-11-19 19:42:45 [INFO    ] Finished processing BAM request
2018-11-19 19:42:45 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:42:45 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561588   8 58441 70965029 A  ERROR'
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:42:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:42:45 [DEBUG   ] spawn_process: at 2018-11-19 19:42:45 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:42:50 [DEBUG   ] Got command STP from MCS: ref #561589
2018-11-19 19:42:50 [INFO    ] Finished processing BAM request
2018-11-19 19:42:50 [INFO    ] Shutdown the adder chain on beam #3
2018-11-19 19:42:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   561589   8 58441 70970025 A  ERROR'
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:42:50 [DEBUG   ] spawn_process: at 2018-11-19 19:42:50 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:42:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:42:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 19:42:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:42:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 19:42:56 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 19:42:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 19:42:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:42:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:42:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:42:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:42:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:42:56 [DEBUG   ] Finished updating board flags in 1.426 seconds
2018-11-19 19:43:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:43:26 [DEBUG   ] Got command BAM from MCS: ref #561594
2018-11-19 19:43:26 [INFO    ] Finished processing BAM request
2018-11-19 19:43:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:43:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561594   8 58441 71006029 A  ERROR'
2018-11-19 19:43:26 [DEBUG   ] Got command BAM from MCS: ref #561595
2018-11-19 19:43:26 [INFO    ] Finished processing BAM request
2018-11-19 19:43:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:43:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561595   8 58441 71006033 A  ERROR'
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 19:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 19:43:26 [DEBUG   ] spawn_process: at 2018-11-19 19:43:26 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 19:43:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:43:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:43:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:43:56 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 19:43:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 19:43:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:43:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:43:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:43:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:43:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:43:56 [DEBUG   ] Finished updating board flags in 1.367 seconds
2018-11-19 19:44:07 [DEBUG   ] Got command BAM from MCS: ref #561596
2018-11-19 19:44:07 [INFO    ] Finished processing BAM request
2018-11-19 19:44:07 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:44:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561596   8 58441 71047033 A  ERROR'
2018-11-19 19:44:07 [DEBUG   ] Got command BAM from MCS: ref #561597
2018-11-19 19:44:07 [INFO    ] Finished processing BAM request
2018-11-19 19:44:07 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:44:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561597   8 58441 71047037 A  ERROR'
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 19:44:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 19:44:07 [DEBUG   ] spawn_process: at 2018-11-19 19:44:07 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 19:44:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:44:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:44:48 [DEBUG   ] Got command BAM from MCS: ref #561599
2018-11-19 19:44:48 [INFO    ] Finished processing BAM request
2018-11-19 19:44:48 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:44:48 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561599   8 58441 71088025 A  ERROR'
2018-11-19 19:44:48 [DEBUG   ] Got command BAM from MCS: ref #561598
2018-11-19 19:44:48 [INFO    ] Finished processing BAM request
2018-11-19 19:44:48 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:44:48 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561598   8 58441 71088029 A  ERROR'
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:44:48 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:44:48 [DEBUG   ] spawn_process: at 2018-11-19 19:44:48 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:44:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:44:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:44:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 19:44:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:44:56 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:44:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:44:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:44:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:44:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:44:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:44:56 [DEBUG   ] Finished updating board flags in 1.217 seconds
2018-11-19 19:44:58 [DEBUG   ] Got command RPT from MCS: ref #561600
2018-11-19 19:44:58 [DEBUG   ] summary = ERROR
2018-11-19 19:44:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561600  13 58441 71098921 A  ERRORERROR'
2018-11-19 19:44:59 [DEBUG   ] Got command RPT from MCS: ref #561601
2018-11-19 19:44:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:44:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561601  77 58441 71099113 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:44:59 [DEBUG   ] Got command RPT from MCS: ref #561602
2018-11-19 19:44:59 [DEBUG   ] summary = ERROR
2018-11-19 19:44:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561602  13 58441 71099317 A  ERRORERROR'
2018-11-19 19:44:59 [DEBUG   ] Got command RPT from MCS: ref #561603
2018-11-19 19:44:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:44:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561603  77 58441 71099521 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:44:59 [DEBUG   ] Got command RPT from MCS: ref #561604
2018-11-19 19:44:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:44:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561604  12 58441 71099725 A  ERRORB¢ff'
2018-11-19 19:45:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:45:29 [DEBUG   ] Got command BAM from MCS: ref #561605
2018-11-19 19:45:29 [INFO    ] Finished processing BAM request
2018-11-19 19:45:29 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:45:29 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561605   8 58441 71129033 A  ERROR'
2018-11-19 19:45:29 [DEBUG   ] Got command BAM from MCS: ref #561606
2018-11-19 19:45:29 [INFO    ] Finished processing BAM request
2018-11-19 19:45:29 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:45:29 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561606   8 58441 71129037 A  ERROR'
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 19:45:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 19:45:29 [DEBUG   ] spawn_process: at 2018-11-19 19:45:29 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 19:45:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 19:45:46 [DEBUG   ] Got command RPT from MCS: ref #561607
2018-11-19 19:45:46 [DEBUG   ] summary = ERROR
2018-11-19 19:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561607  13 58441 71146485 A  ERRORERROR'
2018-11-19 19:45:46 [DEBUG   ] Got command RPT from MCS: ref #561608
2018-11-19 19:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561608  77 58441 71146486 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:45:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 19:45:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:45:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 19:45:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 19:45:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:45:56 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:45:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:45:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:45:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:45:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:45:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:45:56 [DEBUG   ] Finished updating board flags in 1.214 seconds
2018-11-19 19:46:10 [DEBUG   ] Got command BAM from MCS: ref #561629
2018-11-19 19:46:10 [INFO    ] Finished processing BAM request
2018-11-19 19:46:10 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:46:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561629   8 58441 71170033 A  ERROR'
2018-11-19 19:46:10 [DEBUG   ] Got command BAM from MCS: ref #561628
2018-11-19 19:46:10 [INFO    ] Finished processing BAM request
2018-11-19 19:46:10 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:46:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561628   8 58441 71170037 A  ERROR'
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 19:46:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 19:46:10 [DEBUG   ] spawn_process: at 2018-11-19 19:46:10 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 19:46:14 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 19:46:34 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 19:46:51 [DEBUG   ] Got command BAM from MCS: ref #561630
2018-11-19 19:46:51 [INFO    ] Finished processing BAM request
2018-11-19 19:46:51 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:46:51 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561630   8 58441 71211029 A  ERROR'
2018-11-19 19:46:51 [DEBUG   ] Got command BAM from MCS: ref #561631
2018-11-19 19:46:51 [INFO    ] Finished processing BAM request
2018-11-19 19:46:51 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:46:51 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561631   8 58441 71211033 A  ERROR'
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 19:46:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 19:46:51 [DEBUG   ] spawn_process: at 2018-11-19 19:46:51 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 19:46:54 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 19:46:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 19:46:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:46:56 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:46:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:46:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:46:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:46:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:46:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:46:56 [DEBUG   ] Finished updating board flags in 1.168 seconds
2018-11-19 19:47:14 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 19:47:32 [DEBUG   ] Got command BAM from MCS: ref #561633
2018-11-19 19:47:32 [INFO    ] Finished processing BAM request
2018-11-19 19:47:32 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:47:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561633   8 58441 71252029 A  ERROR'
2018-11-19 19:47:32 [DEBUG   ] Got command BAM from MCS: ref #561632
2018-11-19 19:47:32 [INFO    ] Finished processing BAM request
2018-11-19 19:47:32 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:47:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561632   8 58441 71252033 A  ERROR'
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:47:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:47:32 [DEBUG   ] spawn_process: at 2018-11-19 19:47:32 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:47:34 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:47:54 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:47:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 19:47:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:47:56 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:47:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 19:47:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:47:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:47:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:47:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:47:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:47:56 [DEBUG   ] Finished updating board flags in 1.030 seconds
2018-11-19 19:48:13 [DEBUG   ] Got command BAM from MCS: ref #561635
2018-11-19 19:48:13 [INFO    ] Finished processing BAM request
2018-11-19 19:48:13 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:48:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561635   8 58441 71293033 A  ERROR'
2018-11-19 19:48:13 [DEBUG   ] Got command BAM from MCS: ref #561634
2018-11-19 19:48:13 [INFO    ] Finished processing BAM request
2018-11-19 19:48:13 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:48:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561634   8 58441 71293037 A  ERROR'
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 19:48:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 19:48:13 [DEBUG   ] spawn_process: at 2018-11-19 19:48:13 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 19:48:14 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:48:34 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:48:54 [DEBUG   ] Got command BAM from MCS: ref #561637
2018-11-19 19:48:54 [INFO    ] Finished processing BAM request
2018-11-19 19:48:54 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:48:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561637   8 58441 71334029 A  ERROR'
2018-11-19 19:48:54 [DEBUG   ] Got command BAM from MCS: ref #561636
2018-11-19 19:48:54 [INFO    ] Finished processing BAM request
2018-11-19 19:48:54 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:48:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561636   8 58441 71334033 A  ERROR'
2018-11-19 19:48:54 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 19:48:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 19:48:54 [DEBUG   ] spawn_process: at 2018-11-19 19:48:54 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 19:48:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:48:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 19:48:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:48:56 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:48:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 19:48:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:48:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:48:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:48:56 [WARNING ] (13, 1, 'bot')
2018-11-19 19:48:56 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:48:56 [DEBUG   ] Finished updating board flags in 0.973 seconds
2018-11-19 19:49:14 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:49:34 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 19:49:35 [DEBUG   ] Got command BAM from MCS: ref #561642
2018-11-19 19:49:35 [INFO    ] Finished processing BAM request
2018-11-19 19:49:35 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:49:35 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561642   8 58441 71375029 A  ERROR'
2018-11-19 19:49:35 [DEBUG   ] Got command BAM from MCS: ref #561643
2018-11-19 19:49:35 [INFO    ] Finished processing BAM request
2018-11-19 19:49:35 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:49:35 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561643   8 58441 71375033 A  ERROR'
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:49:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:49:35 [DEBUG   ] spawn_process: at 2018-11-19 19:49:35 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:49:54 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 19:49:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:49:57 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:49:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:49:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:49:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:49:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:49:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:49:57 [DEBUG   ] Finished updating board flags in 1.418 seconds
2018-11-19 19:49:58 [DEBUG   ] Got command RPT from MCS: ref #561646
2018-11-19 19:49:58 [DEBUG   ] summary = ERROR
2018-11-19 19:49:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561646  13 58441 71398998 A  ERRORERROR'
2018-11-19 19:49:59 [DEBUG   ] Got command RPT from MCS: ref #561647
2018-11-19 19:49:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:49:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561647  77 58441 71399201 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:49:59 [DEBUG   ] Got command RPT from MCS: ref #561648
2018-11-19 19:49:59 [DEBUG   ] summary = ERROR
2018-11-19 19:49:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561648  13 58441 71399405 A  ERRORERROR'
2018-11-19 19:49:59 [DEBUG   ] Got command RPT from MCS: ref #561650
2018-11-19 19:49:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:49:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561650  77 58441 71399609 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:49:59 [DEBUG   ] Got command RPT from MCS: ref #561652
2018-11-19 19:49:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:49:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561652  12 58441 71399802 A  ERRORB¢ff'
2018-11-19 19:50:14 [DEBUG   ] Finished updating ant stats in 0.399 seconds
2018-11-19 19:50:16 [DEBUG   ] Got command BAM from MCS: ref #561656
2018-11-19 19:50:16 [INFO    ] Finished processing BAM request
2018-11-19 19:50:16 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:50:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561656   8 58441 71416030 A  ERROR'
2018-11-19 19:50:16 [DEBUG   ] Got command BAM from MCS: ref #561657
2018-11-19 19:50:16 [INFO    ] Finished processing BAM request
2018-11-19 19:50:16 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:50:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561657   8 58441 71416034 A  ERROR'
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 19:50:16 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 19:50:16 [DEBUG   ] spawn_process: at 2018-11-19 19:50:16 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 19:50:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 19:50:46 [DEBUG   ] Got command RPT from MCS: ref #561658
2018-11-19 19:50:46 [DEBUG   ] summary = ERROR
2018-11-19 19:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561658  13 58441 71446493 A  ERRORERROR'
2018-11-19 19:50:46 [DEBUG   ] Got command RPT from MCS: ref #561659
2018-11-19 19:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561659  77 58441 71446494 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:50:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 19:50:56 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 19:50:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 19:50:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:50:57 [DEBUG   ] Got command BAM from MCS: ref #561676
2018-11-19 19:50:57 [INFO    ] Finished processing BAM request
2018-11-19 19:50:57 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:50:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561676   8 58441 71457034 A  ERROR'
2018-11-19 19:50:57 [DEBUG   ] Got command BAM from MCS: ref #561675
2018-11-19 19:50:57 [INFO    ] Finished processing BAM request
2018-11-19 19:50:57 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:50:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561675   8 58441 71457038 A  ERROR'
2018-11-19 19:50:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 19:50:57 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:50:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 19:50:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:50:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:50:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:50:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:50:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:50:57 [DEBUG   ] Finished updating board flags in 1.414 seconds
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:50:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:50:57 [DEBUG   ] spawn_process: at 2018-11-19 19:50:57 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:51:15 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 19:51:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 19:51:38 [DEBUG   ] Got command BAM from MCS: ref #561678
2018-11-19 19:51:38 [INFO    ] Finished processing BAM request
2018-11-19 19:51:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:51:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561678   8 58441 71498034 A  ERROR'
2018-11-19 19:51:38 [DEBUG   ] Got command BAM from MCS: ref #561677
2018-11-19 19:51:38 [INFO    ] Finished processing BAM request
2018-11-19 19:51:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:51:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561677   8 58441 71498038 A  ERROR'
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 19:51:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 19:51:38 [DEBUG   ] spawn_process: at 2018-11-19 19:51:38 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 19:51:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 19:51:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 19:51:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 19:51:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:51:57 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:51:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 19:51:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:51:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:51:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:51:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:51:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:51:57 [DEBUG   ] Finished updating board flags in 1.337 seconds
2018-11-19 19:52:15 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 19:52:19 [DEBUG   ] Got command BAM from MCS: ref #561684
2018-11-19 19:52:19 [INFO    ] Finished processing BAM request
2018-11-19 19:52:19 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:52:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561684   8 58441 71539025 A  ERROR'
2018-11-19 19:52:19 [DEBUG   ] Got command BAM from MCS: ref #561683
2018-11-19 19:52:19 [INFO    ] Finished processing BAM request
2018-11-19 19:52:19 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:52:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561683   8 58441 71539029 A  ERROR'
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 19:52:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 19:52:19 [DEBUG   ] spawn_process: at 2018-11-19 19:52:19 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 19:52:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 19:52:55 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 19:52:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 19:52:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:52:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 19:52:57 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:52:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 19:52:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:52:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:52:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:52:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:52:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:52:57 [DEBUG   ] Finished updating board flags in 1.201 seconds
2018-11-19 19:53:00 [DEBUG   ] Got command BAM from MCS: ref #561685
2018-11-19 19:53:00 [INFO    ] Finished processing BAM request
2018-11-19 19:53:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:53:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561685   8 58441 71580025 A  ERROR'
2018-11-19 19:53:00 [DEBUG   ] Got command BAM from MCS: ref #561686
2018-11-19 19:53:00 [INFO    ] Finished processing BAM request
2018-11-19 19:53:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:53:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561686   8 58441 71580029 A  ERROR'
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 19:53:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 19:53:00 [DEBUG   ] spawn_process: at 2018-11-19 19:53:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 19:53:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 19:53:35 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 19:53:40 [DEBUG   ] Got command BAM from MCS: ref #561688
2018-11-19 19:53:40 [INFO    ] Finished processing BAM request
2018-11-19 19:53:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:53:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561688   8 58441 71620025 A  ERROR'
2018-11-19 19:53:40 [DEBUG   ] Got command BAM from MCS: ref #561687
2018-11-19 19:53:40 [INFO    ] Finished processing BAM request
2018-11-19 19:53:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:53:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561687   8 58441 71620029 A  ERROR'
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 19:53:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 19:53:40 [DEBUG   ] spawn_process: at 2018-11-19 19:53:40 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 19:53:55 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 19:53:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:53:56 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 19:53:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:53:57 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:53:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 19:53:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:53:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:53:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:53:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:53:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:53:57 [DEBUG   ] Finished updating board flags in 1.219 seconds
2018-11-19 19:54:15 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 19:54:20 [DEBUG   ] Got command BAM from MCS: ref #561689
2018-11-19 19:54:20 [INFO    ] Finished processing BAM request
2018-11-19 19:54:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:54:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561689   8 58441 71660025 A  ERROR'
2018-11-19 19:54:20 [DEBUG   ] Got command BAM from MCS: ref #561690
2018-11-19 19:54:20 [INFO    ] Finished processing BAM request
2018-11-19 19:54:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:54:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561690   8 58441 71660029 A  ERROR'
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 19:54:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 19:54:20 [DEBUG   ] spawn_process: at 2018-11-19 19:54:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 19:54:35 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 19:54:55 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 19:54:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 19:54:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 19:54:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:54:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 19:54:57 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:54:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:54:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:54:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:54:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:54:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:54:57 [DEBUG   ] Finished updating board flags in 1.190 seconds
2018-11-19 19:54:59 [DEBUG   ] Got command RPT from MCS: ref #561693
2018-11-19 19:54:59 [DEBUG   ] summary = ERROR
2018-11-19 19:54:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561693  13 58441 71699085 A  ERRORERROR'
2018-11-19 19:54:59 [DEBUG   ] Got command RPT from MCS: ref #561694
2018-11-19 19:54:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:54:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561694  77 58441 71699289 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:54:59 [DEBUG   ] Got command RPT from MCS: ref #561695
2018-11-19 19:54:59 [DEBUG   ] summary = ERROR
2018-11-19 19:54:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561695  13 58441 71699493 A  ERRORERROR'
2018-11-19 19:54:59 [DEBUG   ] Got command RPT from MCS: ref #561696
2018-11-19 19:54:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:54:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561696  77 58441 71699685 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:54:59 [DEBUG   ] Got command RPT from MCS: ref #561697
2018-11-19 19:54:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:54:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561697  12 58441 71699889 A  ERRORB¢ff'
2018-11-19 19:55:00 [DEBUG   ] Got command BAM from MCS: ref #561692
2018-11-19 19:55:00 [INFO    ] Finished processing BAM request
2018-11-19 19:55:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:55:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561692   8 58441 71700025 A  ERROR'
2018-11-19 19:55:00 [DEBUG   ] Got command BAM from MCS: ref #561691
2018-11-19 19:55:00 [INFO    ] Finished processing BAM request
2018-11-19 19:55:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:55:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561691   8 58441 71700029 A  ERROR'
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:55:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:55:00 [DEBUG   ] spawn_process: at 2018-11-19 19:55:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:55:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 19:55:35 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 19:55:40 [DEBUG   ] Got command BAM from MCS: ref #561702
2018-11-19 19:55:40 [INFO    ] Finished processing BAM request
2018-11-19 19:55:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:55:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561702   8 58441 71740025 A  ERROR'
2018-11-19 19:55:40 [DEBUG   ] Got command BAM from MCS: ref #561703
2018-11-19 19:55:40 [INFO    ] Finished processing BAM request
2018-11-19 19:55:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:55:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561703   8 58441 71740029 A  ERROR'
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:55:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:55:40 [DEBUG   ] spawn_process: at 2018-11-19 19:55:40 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:55:46 [DEBUG   ] Got command RPT from MCS: ref #561704
2018-11-19 19:55:46 [DEBUG   ] summary = ERROR
2018-11-19 19:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561704  13 58441 71746489 A  ERRORERROR'
2018-11-19 19:55:46 [DEBUG   ] Got command RPT from MCS: ref #561705
2018-11-19 19:55:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561705  77 58441 71746490 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:55:55 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:55:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 19:55:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 19:55:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:55:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 19:55:57 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 19:55:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 19:55:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:55:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:55:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:55:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:55:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:55:57 [DEBUG   ] Finished updating board flags in 1.013 seconds
2018-11-19 19:56:15 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 19:56:20 [DEBUG   ] Got command BAM from MCS: ref #561721
2018-11-19 19:56:20 [INFO    ] Finished processing BAM request
2018-11-19 19:56:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:56:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561721   8 58441 71780025 A  ERROR'
2018-11-19 19:56:20 [DEBUG   ] Got command BAM from MCS: ref #561722
2018-11-19 19:56:20 [INFO    ] Finished processing BAM request
2018-11-19 19:56:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:56:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561722   8 58441 71780029 A  ERROR'
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 19:56:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 19:56:20 [DEBUG   ] spawn_process: at 2018-11-19 19:56:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 19:56:35 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 19:56:55 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 19:56:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:56:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 19:56:57 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:56:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 19:56:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:56:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:56:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:56:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:56:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:56:57 [DEBUG   ] Finished updating board flags in 1.037 seconds
2018-11-19 19:57:00 [DEBUG   ] Got command BAM from MCS: ref #561724
2018-11-19 19:57:00 [INFO    ] Finished processing BAM request
2018-11-19 19:57:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:57:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561724   8 58441 71820025 A  ERROR'
2018-11-19 19:57:00 [DEBUG   ] Got command BAM from MCS: ref #561723
2018-11-19 19:57:00 [INFO    ] Finished processing BAM request
2018-11-19 19:57:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:57:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561723   8 58441 71820029 A  ERROR'
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 19:57:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 19:57:00 [DEBUG   ] spawn_process: at 2018-11-19 19:57:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 19:57:15 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 19:57:35 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 19:57:40 [DEBUG   ] Got command BAM from MCS: ref #561726
2018-11-19 19:57:40 [INFO    ] Finished processing BAM request
2018-11-19 19:57:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:57:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561726   8 58441 71860029 A  ERROR'
2018-11-19 19:57:40 [DEBUG   ] Got command BAM from MCS: ref #561725
2018-11-19 19:57:40 [INFO    ] Finished processing BAM request
2018-11-19 19:57:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:57:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561725   8 58441 71860033 A  ERROR'
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 19:57:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 19:57:40 [DEBUG   ] spawn_process: at 2018-11-19 19:57:40 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 19:57:55 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:57:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:57:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 19:57:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:57:57 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 19:57:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 19:57:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 19:57:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:57:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:57:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:57:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:57:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:57:57 [DEBUG   ] Finished updating board flags in 1.481 seconds
2018-11-19 19:58:15 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:58:20 [DEBUG   ] Got command BAM from MCS: ref #561731
2018-11-19 19:58:20 [INFO    ] Finished processing BAM request
2018-11-19 19:58:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:58:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561731   8 58441 71900033 A  ERROR'
2018-11-19 19:58:20 [DEBUG   ] Got command BAM from MCS: ref #561732
2018-11-19 19:58:20 [INFO    ] Finished processing BAM request
2018-11-19 19:58:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:58:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561732   8 58441 71900037 A  ERROR'
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 19:58:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 19:58:20 [DEBUG   ] spawn_process: at 2018-11-19 19:58:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 19:58:35 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 19:58:55 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 19:58:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:58:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 19:58:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:58:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 19:58:57 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 19:58:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 19:58:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:58:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:58:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:58:57 [WARNING ] (13, 1, 'bot')
2018-11-19 19:58:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:58:57 [DEBUG   ] Finished updating board flags in 1.315 seconds
2018-11-19 19:59:00 [DEBUG   ] Got command BAM from MCS: ref #561734
2018-11-19 19:59:00 [INFO    ] Finished processing BAM request
2018-11-19 19:59:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:59:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561734   8 58441 71940033 A  ERROR'
2018-11-19 19:59:00 [DEBUG   ] Got command BAM from MCS: ref #561733
2018-11-19 19:59:00 [INFO    ] Finished processing BAM request
2018-11-19 19:59:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:59:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561733   8 58441 71940037 A  ERROR'
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 19:59:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 19:59:00 [DEBUG   ] spawn_process: at 2018-11-19 19:59:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 19:59:15 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 19:59:35 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 19:59:40 [DEBUG   ] Got command BAM from MCS: ref #561736
2018-11-19 19:59:40 [INFO    ] Finished processing BAM request
2018-11-19 19:59:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:59:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561736   8 58441 71980029 A  ERROR'
2018-11-19 19:59:40 [DEBUG   ] Got command BAM from MCS: ref #561735
2018-11-19 19:59:40 [INFO    ] Finished processing BAM request
2018-11-19 19:59:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 19:59:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561735   8 58441 71980033 A  ERROR'
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 19:59:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 19:59:40 [DEBUG   ] spawn_process: at 2018-11-19 19:59:40 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 19:59:55 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 19:59:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 19:59:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 19:59:58 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 19:59:58 [WARNING ] (13, 1, 'bot')
2018-11-19 19:59:58 [WARNING ] (13, 1, 'bot')
2018-11-19 19:59:58 [WARNING ] (13, 1, 'bot')
2018-11-19 19:59:58 [WARNING ] (13, 1, 'bot')
2018-11-19 19:59:58 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 19:59:58 [DEBUG   ] Finished updating board flags in 1.778 seconds
2018-11-19 19:59:59 [DEBUG   ] Got command RPT from MCS: ref #561739
2018-11-19 19:59:59 [DEBUG   ] summary = ERROR
2018-11-19 19:59:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561739  13 58441 71999169 A  ERRORERROR'
2018-11-19 19:59:59 [DEBUG   ] Got command RPT from MCS: ref #561740
2018-11-19 19:59:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:59:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561740  77 58441 71999373 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:59:59 [DEBUG   ] Got command RPT from MCS: ref #561741
2018-11-19 19:59:59 [DEBUG   ] summary = ERROR
2018-11-19 19:59:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561741  13 58441 71999577 A  ERRORERROR'
2018-11-19 19:59:59 [DEBUG   ] Got command RPT from MCS: ref #561743
2018-11-19 19:59:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 19:59:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561743  77 58441 71999769 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 19:59:59 [DEBUG   ] Got command RPT from MCS: ref #561745
2018-11-19 19:59:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 19:59:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561745  12 58441 71999973 A  ERRORB¡ff'
2018-11-19 20:00:15 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 20:00:20 [DEBUG   ] Got command BAM from MCS: ref #561750
2018-11-19 20:00:20 [INFO    ] Finished processing BAM request
2018-11-19 20:00:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:00:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561750   8 58441 72020033 A  ERROR'
2018-11-19 20:00:20 [DEBUG   ] Got command BAM from MCS: ref #561749
2018-11-19 20:00:20 [INFO    ] Finished processing BAM request
2018-11-19 20:00:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:00:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561749   8 58441 72020037 A  ERROR'
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 20:00:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 20:00:20 [DEBUG   ] spawn_process: at 2018-11-19 20:00:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 20:00:35 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 20:00:46 [DEBUG   ] Got command RPT from MCS: ref #561751
2018-11-19 20:00:46 [DEBUG   ] summary = ERROR
2018-11-19 20:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561751  13 58441 72046489 A  ERRORERROR'
2018-11-19 20:00:46 [DEBUG   ] Got command RPT from MCS: ref #561752
2018-11-19 20:00:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561752  77 58441 72046490 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:00:55 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 20:00:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:00:57 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:00:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:00:57 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 20:00:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:00:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:00:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:00:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:00:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:00:57 [DEBUG   ] Finished updating board flags in 1.026 seconds
2018-11-19 20:01:00 [DEBUG   ] Got command BAM from MCS: ref #561769
2018-11-19 20:01:00 [INFO    ] Finished processing BAM request
2018-11-19 20:01:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:01:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561769   8 58441 72060025 A  ERROR'
2018-11-19 20:01:00 [DEBUG   ] Got command BAM from MCS: ref #561768
2018-11-19 20:01:00 [INFO    ] Finished processing BAM request
2018-11-19 20:01:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:01:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561768   8 58441 72060029 A  ERROR'
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 20:01:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 20:01:00 [DEBUG   ] spawn_process: at 2018-11-19 20:01:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 20:01:15 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 20:01:35 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 20:01:40 [DEBUG   ] Got command BAM from MCS: ref #561774
2018-11-19 20:01:40 [INFO    ] Finished processing BAM request
2018-11-19 20:01:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:01:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561774   8 58441 72100033 A  ERROR'
2018-11-19 20:01:40 [DEBUG   ] Got command BAM from MCS: ref #561775
2018-11-19 20:01:40 [INFO    ] Finished processing BAM request
2018-11-19 20:01:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:01:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561775   8 58441 72100037 A  ERROR'
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 20:01:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 20:01:40 [DEBUG   ] spawn_process: at 2018-11-19 20:01:40 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 20:01:55 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 20:01:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:01:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:01:57 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 20:01:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:01:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:01:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:01:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:01:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:01:57 [DEBUG   ] Finished updating board flags in 1.020 seconds
2018-11-19 20:02:15 [DEBUG   ] Finished updating ant stats in 0.994 seconds
2018-11-19 20:02:20 [DEBUG   ] Got command BAM from MCS: ref #561776
2018-11-19 20:02:20 [INFO    ] Finished processing BAM request
2018-11-19 20:02:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:02:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561776   8 58441 72140025 A  ERROR'
2018-11-19 20:02:20 [DEBUG   ] Got command BAM from MCS: ref #561777
2018-11-19 20:02:20 [INFO    ] Finished processing BAM request
2018-11-19 20:02:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:02:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561777   8 58441 72140029 A  ERROR'
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 20:02:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 20:02:20 [DEBUG   ] spawn_process: at 2018-11-19 20:02:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 20:02:35 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:02:55 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 20:02:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:02:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:02:57 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 20:02:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:02:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:02:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:02:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:02:57 [WARNING ] (13, 1, 'bot')
2018-11-19 20:02:57 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:02:57 [DEBUG   ] Finished updating board flags in 0.994 seconds
2018-11-19 20:03:00 [DEBUG   ] Got command BAM from MCS: ref #561779
2018-11-19 20:03:00 [INFO    ] Finished processing BAM request
2018-11-19 20:03:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:03:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561779   8 58441 72180025 A  ERROR'
2018-11-19 20:03:00 [DEBUG   ] Got command BAM from MCS: ref #561778
2018-11-19 20:03:00 [INFO    ] Finished processing BAM request
2018-11-19 20:03:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:03:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561778   8 58441 72180029 A  ERROR'
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 20:03:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 20:03:00 [DEBUG   ] spawn_process: at 2018-11-19 20:03:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 20:03:15 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:03:35 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:03:40 [DEBUG   ] Got command BAM from MCS: ref #561780
2018-11-19 20:03:40 [INFO    ] Finished processing BAM request
2018-11-19 20:03:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:03:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561780   8 58441 72220029 A  ERROR'
2018-11-19 20:03:40 [DEBUG   ] Got command BAM from MCS: ref #561781
2018-11-19 20:03:40 [INFO    ] Finished processing BAM request
2018-11-19 20:03:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:03:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561781   8 58441 72220033 A  ERROR'
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 20:03:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 20:03:40 [DEBUG   ] spawn_process: at 2018-11-19 20:03:40 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 20:03:55 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:03:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:03:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:03:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:03:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:03:58 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 20:03:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:03:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:03:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:03:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:03:58 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:03:58 [DEBUG   ] Finished updating board flags in 1.354 seconds
2018-11-19 20:04:15 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:04:20 [DEBUG   ] Got command BAM from MCS: ref #561787
2018-11-19 20:04:20 [INFO    ] Finished processing BAM request
2018-11-19 20:04:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:04:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561787   8 58441 72260025 A  ERROR'
2018-11-19 20:04:20 [DEBUG   ] Got command BAM from MCS: ref #561786
2018-11-19 20:04:20 [INFO    ] Finished processing BAM request
2018-11-19 20:04:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:04:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561786   8 58441 72260029 A  ERROR'
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 20:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 20:04:20 [DEBUG   ] spawn_process: at 2018-11-19 20:04:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 20:04:35 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 20:04:55 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 20:04:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:04:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:04:58 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 20:04:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:04:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:04:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:04:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:04:58 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:04:58 [DEBUG   ] Finished updating board flags in 1.195 seconds
2018-11-19 20:04:59 [DEBUG   ] Got command RPT from MCS: ref #561790
2018-11-19 20:04:59 [DEBUG   ] summary = ERROR
2018-11-19 20:04:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561790  13 58441 72299257 A  ERRORERROR'
2018-11-19 20:04:59 [DEBUG   ] Got command RPT from MCS: ref #561791
2018-11-19 20:04:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:04:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561791  77 58441 72299461 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:04:59 [DEBUG   ] Got command RPT from MCS: ref #561792
2018-11-19 20:04:59 [DEBUG   ] summary = ERROR
2018-11-19 20:04:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561792  13 58441 72299665 A  ERRORERROR'
2018-11-19 20:04:59 [DEBUG   ] Got command RPT from MCS: ref #561793
2018-11-19 20:04:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:04:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561793  77 58441 72299857 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:05:00 [DEBUG   ] Got command BAM from MCS: ref #561788
2018-11-19 20:05:00 [INFO    ] Finished processing BAM request
2018-11-19 20:05:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:05:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561788   8 58441 72300029 A  ERROR'
2018-11-19 20:05:00 [DEBUG   ] Got command BAM from MCS: ref #561789
2018-11-19 20:05:00 [INFO    ] Finished processing BAM request
2018-11-19 20:05:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:05:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561789   8 58441 72300033 A  ERROR'
2018-11-19 20:05:00 [DEBUG   ] Got command RPT from MCS: ref #561794
2018-11-19 20:05:00 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:05:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561794  12 58441 72300057 A  ERRORB¡ff'
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 20:05:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 20:05:00 [DEBUG   ] spawn_process: at 2018-11-19 20:05:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 20:05:15 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:05:35 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:05:40 [DEBUG   ] Got command BAM from MCS: ref #561795
2018-11-19 20:05:40 [INFO    ] Finished processing BAM request
2018-11-19 20:05:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:05:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561795   8 58441 72340037 A  ERROR'
2018-11-19 20:05:40 [DEBUG   ] Got command BAM from MCS: ref #561796
2018-11-19 20:05:40 [INFO    ] Finished processing BAM request
2018-11-19 20:05:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:05:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561796   8 58441 72340045 A  ERROR'
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 20:05:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 20:05:40 [DEBUG   ] spawn_process: at 2018-11-19 20:05:40 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 20:05:46 [DEBUG   ] Got command RPT from MCS: ref #561797
2018-11-19 20:05:46 [DEBUG   ] summary = ERROR
2018-11-19 20:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561797  13 58441 72346489 A  ERRORERROR'
2018-11-19 20:05:46 [DEBUG   ] Got command RPT from MCS: ref #561798
2018-11-19 20:05:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561798  77 58441 72346490 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:05:55 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 20:05:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:05:57 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:05:57 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:05:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:05:58 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 20:05:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:05:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:05:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:05:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:05:58 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:05:58 [DEBUG   ] Finished updating board flags in 1.038 seconds
2018-11-19 20:06:15 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 20:06:20 [DEBUG   ] Got command BAM from MCS: ref #561814
2018-11-19 20:06:20 [INFO    ] Finished processing BAM request
2018-11-19 20:06:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:06:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561814   8 58441 72380033 A  ERROR'
2018-11-19 20:06:20 [DEBUG   ] Got command BAM from MCS: ref #561815
2018-11-19 20:06:20 [INFO    ] Finished processing BAM request
2018-11-19 20:06:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:06:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561815   8 58441 72380037 A  ERROR'
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 20:06:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 20:06:20 [DEBUG   ] spawn_process: at 2018-11-19 20:06:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 20:06:35 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:06:55 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:06:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:06:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:06:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:06:58 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 20:06:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:06:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:06:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:06:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:06:58 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:06:58 [DEBUG   ] Finished updating board flags in 1.475 seconds
2018-11-19 20:07:00 [DEBUG   ] Got command BAM from MCS: ref #561816
2018-11-19 20:07:00 [INFO    ] Finished processing BAM request
2018-11-19 20:07:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:07:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561816   8 58441 72420029 A  ERROR'
2018-11-19 20:07:00 [DEBUG   ] Got command BAM from MCS: ref #561817
2018-11-19 20:07:00 [INFO    ] Finished processing BAM request
2018-11-19 20:07:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:07:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561817   8 58441 72420033 A  ERROR'
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 20:07:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 20:07:00 [DEBUG   ] spawn_process: at 2018-11-19 20:07:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 20:07:15 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 20:07:35 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:07:40 [DEBUG   ] Got command BAM from MCS: ref #561823
2018-11-19 20:07:40 [INFO    ] Finished processing BAM request
2018-11-19 20:07:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:07:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561823   8 58441 72460029 A  ERROR'
2018-11-19 20:07:40 [DEBUG   ] Got command BAM from MCS: ref #561822
2018-11-19 20:07:40 [INFO    ] Finished processing BAM request
2018-11-19 20:07:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:07:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561822   8 58441 72460033 A  ERROR'
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 20:07:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 20:07:40 [DEBUG   ] spawn_process: at 2018-11-19 20:07:40 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 20:07:55 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:07:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:07:59 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:07:59 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 20:07:59 [WARNING ] (13, 1, 'bot')
2018-11-19 20:07:59 [WARNING ] (13, 1, 'bot')
2018-11-19 20:07:59 [WARNING ] (13, 1, 'bot')
2018-11-19 20:07:59 [WARNING ] (13, 1, 'bot')
2018-11-19 20:07:59 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:07:59 [DEBUG   ] Finished updating board flags in 1.964 seconds
2018-11-19 20:08:15 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 20:08:20 [DEBUG   ] Got command BAM from MCS: ref #561825
2018-11-19 20:08:20 [INFO    ] Finished processing BAM request
2018-11-19 20:08:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:08:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561825   8 58441 72500029 A  ERROR'
2018-11-19 20:08:20 [DEBUG   ] Got command BAM from MCS: ref #561824
2018-11-19 20:08:20 [INFO    ] Finished processing BAM request
2018-11-19 20:08:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:08:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561824   8 58441 72500033 A  ERROR'
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 20:08:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 20:08:20 [DEBUG   ] spawn_process: at 2018-11-19 20:08:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 20:08:35 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:08:55 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:08:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:08:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:08:58 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 20:08:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:08:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:08:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:08:58 [WARNING ] (13, 1, 'bot')
2018-11-19 20:08:58 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:08:58 [DEBUG   ] Finished updating board flags in 1.178 seconds
2018-11-19 20:09:00 [DEBUG   ] Got command BAM from MCS: ref #561827
2018-11-19 20:09:00 [INFO    ] Finished processing BAM request
2018-11-19 20:09:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:09:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561827   8 58441 72540025 A  ERROR'
2018-11-19 20:09:00 [DEBUG   ] Got command BAM from MCS: ref #561826
2018-11-19 20:09:00 [INFO    ] Finished processing BAM request
2018-11-19 20:09:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:09:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561826   8 58441 72540029 A  ERROR'
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 20:09:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 20:09:00 [DEBUG   ] spawn_process: at 2018-11-19 20:09:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa27: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa24: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa23: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa18: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa10: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa32: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa11: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa30: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa12: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa8: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa13: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa26: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa14: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa22: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa20: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa1: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa5: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa25: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa35: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa19: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa21: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa6: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa7: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa33: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa15: connect: No route to host

2018-11-19 20:09:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa28: connect: No route to host

2018-11-19 20:09:38 [DEBUG   ] Finished updating ant stats in 24.049 seconds
2018-11-19 20:09:40 [DEBUG   ] Got command BAM from MCS: ref #561828
2018-11-19 20:09:40 [INFO    ] Finished processing BAM request
2018-11-19 20:09:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:09:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561828   8 58441 72580029 A  ERROR'
2018-11-19 20:09:40 [DEBUG   ] Got command BAM from MCS: ref #561829
2018-11-19 20:09:40 [INFO    ] Finished processing BAM request
2018-11-19 20:09:40 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:09:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561829   8 58441 72580033 A  ERROR'
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 20:09:40 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 20:09:40 [DEBUG   ] spawn_process: at 2018-11-19 20:09:40 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa27: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa24: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa23: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa18: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa10: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa32: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa11: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa30: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa12: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa8: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa13: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa26: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa14: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa22: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa20: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa1: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa5: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa25: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa35: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa19: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa21: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa6: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa7: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa33: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa15: connect: No route to host

2018-11-19 20:09:41 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:09:41 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa28: connect: No route to host

2018-11-19 20:09:41 [DEBUG   ] Finished updating ant stats in 3.052 seconds
2018-11-19 20:09:59 [DEBUG   ] Got command RPT from MCS: ref #561832
2018-11-19 20:09:59 [DEBUG   ] summary = ERROR
2018-11-19 20:09:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561832  13 58441 72599345 A  ERRORERROR'
2018-11-19 20:09:59 [DEBUG   ] Got command RPT from MCS: ref #561833
2018-11-19 20:09:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:09:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561833  77 58441 72599549 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:09:59 [DEBUG   ] Got command RPT from MCS: ref #561834
2018-11-19 20:09:59 [DEBUG   ] summary = ERROR
2018-11-19 20:09:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561834  13 58441 72599753 A  ERRORERROR'
2018-11-19 20:09:59 [DEBUG   ] Got command RPT from MCS: ref #561836
2018-11-19 20:09:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:09:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561836  77 58441 72599945 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:10:00 [DEBUG   ] Got command RPT from MCS: ref #561837
2018-11-19 20:10:00 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:10:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561837  12 58441 72600149 A  ERRORB¢ff'
2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa27: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa24: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa23: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa18: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa10: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa32: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa11: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa30: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa12: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa8: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa13: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa26: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa14: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa22: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa20: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa1: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa5: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa25: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa35: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa19: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa21: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa6: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa7: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa33: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa15: connect: No route to host

2018-11-19 20:10:01 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:01 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa28: connect: No route to host

2018-11-19 20:10:01 [DEBUG   ] Finished updating ant stats in 3.053 seconds
2018-11-19 20:10:09 [WARNING ] MonitorFlags: no access to lwa16
2018-11-19 20:10:09 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa16 /lwa/software/flags ALL 1
2018-11-19 20:10:09 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa16: connect: No route to host

2018-11-19 20:10:20 [DEBUG   ] Got command BAM from MCS: ref #561847
2018-11-19 20:10:20 [INFO    ] Finished processing BAM request
2018-11-19 20:10:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:10:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561847   8 58441 72620025 A  ERROR'
2018-11-19 20:10:20 [DEBUG   ] Got command BAM from MCS: ref #561846
2018-11-19 20:10:20 [INFO    ] Finished processing BAM request
2018-11-19 20:10:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 20:10:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561846   8 58441 72620029 A  ERROR'
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 20:10:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 20:10:20 [DEBUG   ] spawn_process: at 2018-11-19 20:10:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa27: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa24: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa23: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa18: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa10: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa32: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa11: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa30: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa12: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa8: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa13: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa26: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa14: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa22: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa20: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa1: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa5: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa25: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa35: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa19: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa21: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa6: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa7: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa33: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa15: connect: Connection refused

2018-11-19 20:10:21 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:21 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa28: connect: Connection refused

2018-11-19 20:10:21 [DEBUG   ] Finished updating ant stats in 3.045 seconds
2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa27
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa27 /lwa/software/flags ALL 2
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa27: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa24
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa24 /lwa/software/flags ALL 3
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa24: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa23
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa23 /lwa/software/flags ALL 4
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa23: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa18
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa18 /lwa/software/flags ALL 5
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa18: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa10
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa10 /lwa/software/flags ALL 6
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa10: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa32
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa32 /lwa/software/flags ALL 7
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa32: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa11
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa11 /lwa/software/flags ALL 8
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa11: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa30
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa30 /lwa/software/flags ALL 9
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa30: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa12
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa12 /lwa/software/flags ALL 10
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa12: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa8
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa8 /lwa/software/flags ALL 11
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa8: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa13
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa13 /lwa/software/flags ALL 12
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa13: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa26
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa26 /lwa/software/flags ALL 13
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa26: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa14
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa14 /lwa/software/flags ALL 14
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa14: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa34
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa34 /lwa/software/flags ALL 1
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa34: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa22
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa22 /lwa/software/flags ALL 2
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa22: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa20
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa20 /lwa/software/flags ALL 3
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa20: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa1
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa1 /lwa/software/flags ALL 4
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa1: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa5
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa5 /lwa/software/flags ALL 5
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa5: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa25
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa25 /lwa/software/flags ALL 6
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa25: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa35
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa35 /lwa/software/flags ALL 7
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa35: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa19
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa19 /lwa/software/flags ALL 8
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa19: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa21
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa21 /lwa/software/flags ALL 9
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa21: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa6
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa6 /lwa/software/flags ALL 10
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa6: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa7
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa7 /lwa/software/flags ALL 11
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa7: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa33
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa33 /lwa/software/flags ALL 12
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa33: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa15
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa15 /lwa/software/flags ALL 13
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa15: connect: Connection refused

2018-11-19 20:10:30 [WARNING ] MonitorFlags: no access to lwa28
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa28 /lwa/software/flags ALL 14
2018-11-19 20:10:30 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa28: connect: Connection refused

2018-11-19 20:10:30 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:10:30 [DEBUG   ] Finished updating board flags in 33.099 seconds
2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa27: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa24: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa23: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa18: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa10: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa32: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa11: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa30: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa12: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa8: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa13: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa26: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa14: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa22: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa20: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa1: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa5: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa25: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa35: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa19: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa21: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa6: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa7: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa33: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa15: connect: Connection refused

2018-11-19 20:10:38 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:38 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa28: connect: Connection refused

2018-11-19 20:10:38 [DEBUG   ] Finished updating ant stats in 0.046 seconds
2018-11-19 20:10:46 [DEBUG   ] Got command RPT from MCS: ref #561848
2018-11-19 20:10:46 [DEBUG   ] summary = ERROR
2018-11-19 20:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561848  13 58441 72646493 A  ERRORERROR'
2018-11-19 20:10:46 [DEBUG   ] Got command RPT from MCS: ref #561849
2018-11-19 20:10:46 [DEBUG   ] info = NUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28
2018-11-19 20:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561849 131 58441 72646494 A  ERRORNUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28'
2018-11-19 20:10:57 [WARNING ] MonitorFlags: no access to lwa16
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa16 /lwa/software/flags ALL 1
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa16: connect: Connection refused

2018-11-19 20:10:57 [WARNING ] MonitorFlags: no access to lwa27
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa27 /lwa/software/flags ALL 2
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa27: connect: Connection refused

2018-11-19 20:10:57 [WARNING ] MonitorFlags: no access to lwa24
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa24 /lwa/software/flags ALL 3
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa24: connect: Connection refused

2018-11-19 20:10:57 [WARNING ] MonitorFlags: no access to lwa23
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa23 /lwa/software/flags ALL 4
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa23: connect: Connection refused

2018-11-19 20:10:57 [WARNING ] MonitorFlags: no access to lwa18
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa18 /lwa/software/flags ALL 5
2018-11-19 20:10:57 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa18: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa10
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa10 /lwa/software/flags ALL 6
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa10: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa32
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa32 /lwa/software/flags ALL 7
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa32: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa11
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa11 /lwa/software/flags ALL 8
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa11: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa30
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa30 /lwa/software/flags ALL 9
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa30: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa12
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa12 /lwa/software/flags ALL 10
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa12: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa8
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa8 /lwa/software/flags ALL 11
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa8: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa13
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa13 /lwa/software/flags ALL 12
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa13: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa26
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa26 /lwa/software/flags ALL 13
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa26: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa14
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa14 /lwa/software/flags ALL 14
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa14: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa34
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa34 /lwa/software/flags ALL 1
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa34: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa22
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa22 /lwa/software/flags ALL 2
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa22: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa20
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa20 /lwa/software/flags ALL 3
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa20: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa1
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa1 /lwa/software/flags ALL 4
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa1: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa5
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa5 /lwa/software/flags ALL 5
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa5: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa25
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa25 /lwa/software/flags ALL 6
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa25: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa35
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa35 /lwa/software/flags ALL 7
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa35: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa19
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa19 /lwa/software/flags ALL 8
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa19: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa21
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa21 /lwa/software/flags ALL 9
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa21: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa6
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa6 /lwa/software/flags ALL 10
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa6: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa7
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa7 /lwa/software/flags ALL 11
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa7: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa33
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa33 /lwa/software/flags ALL 12
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa33: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa15
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa15 /lwa/software/flags ALL 13
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa15: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] MonitorFlags: no access to lwa28
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command was /usr/local/bin/lwa28 /lwa/software/flags ALL 14
2018-11-19 20:10:58 [WARNING ] MonitorFlags: last command output as /usr/local/bin/lwa28: connect: Connection refused

2018-11-19 20:10:58 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:10:58 [DEBUG   ] Finished updating board flags in 0.157 seconds
2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa27: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa24: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa23: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa18: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa10: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa32: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa11: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa30: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa12: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa8: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa13: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa26: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa14: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa22: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa20: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa1: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa5: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa25: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa35: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa19: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa21: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa6: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa7: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa33: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa15: connect: Connection refused

2018-11-19 20:10:58 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:10:58 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa28: connect: Connection refused

2018-11-19 20:10:58 [DEBUG   ] Finished updating ant stats in 0.051 seconds
2018-11-19 20:11:00 [DEBUG   ] Got command BAM from MCS: ref #561865
2018-11-19 20:11:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:11:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561865  75 58441 72660021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:11:00 [DEBUG   ] Got command BAM from MCS: ref #561866
2018-11-19 20:11:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:11:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561866  75 58441 72660022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:11:18 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:11:18 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa24: connect: Connection refused

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:11:18 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:11:18 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa26: connect: Connection refused

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such file or directory

2018-11-19 20:11:18 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:11:18 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:11:18 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa1: connect: Connection refused

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:18 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:19 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:11:19 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:11:19 [WARNING ] AntennaStats: last command output as 
2018-11-19 20:11:19 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:19 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:19 [WARNING ] AntennaStats: no access to lwa28
2018-11-19 20:11:19 [WARNING ] AntennaStats: last command was /usr/local/bin/lwa28 /lwa/software/amplitude
2018-11-19 20:11:19 [WARNING ] AntennaStats: last command output as /usr/local/bin/lwa15: connect: Connection refused

2018-11-19 20:11:19 [WARNING ] AntennaStats: command returned 1; /lwa/software/amplitude: open "/dev/ebc1": No such device or address

2018-11-19 20:11:19 [DEBUG   ] Finished updating ant stats in 0.496 seconds
2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:38 [DEBUG   ] Finished updating ant stats in 0.074 seconds
2018-11-19 20:11:40 [DEBUG   ] Got command BAM from MCS: ref #561867
2018-11-19 20:11:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:11:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561867  75 58441 72700021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:11:40 [DEBUG   ] Got command BAM from MCS: ref #561868
2018-11-19 20:11:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:11:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561868  75 58441 72700022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:11:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:11:58 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:11:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:12:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:12:00 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:12:00 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:12:00 [DEBUG   ] Finished updating board flags in 2.224 seconds
2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:19 [DEBUG   ] Finished updating ant stats in 0.068 seconds
2018-11-19 20:12:20 [DEBUG   ] Got command BAM from MCS: ref #561870
2018-11-19 20:12:20 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:12:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561870  75 58441 72740025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:12:20 [DEBUG   ] Got command BAM from MCS: ref #561869
2018-11-19 20:12:20 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:12:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561869  75 58441 72740026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:39 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:12:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:12:59 [DEBUG   ] Finished updating ant stats in 0.076 seconds
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:12:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:13:00 [DEBUG   ] Got command BAM from MCS: ref #561872
2018-11-19 20:13:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:13:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561872  75 58441 72780025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:13:00 [DEBUG   ] Got command BAM from MCS: ref #561871
2018-11-19 20:13:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:13:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561871  75 58441 72780026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:13:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:13:00 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:13:00 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:13:00 [DEBUG   ] Finished updating board flags in 2.198 seconds
2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:19 [DEBUG   ] Finished updating ant stats in 0.071 seconds
2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:39 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:13:40 [DEBUG   ] Got command BAM from MCS: ref #561878
2018-11-19 20:13:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:13:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561878  75 58441 72820021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:13:40 [DEBUG   ] Got command BAM from MCS: ref #561877
2018-11-19 20:13:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:13:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561877  75 58441 72820022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:13:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:13:59 [DEBUG   ] Finished updating ant stats in 0.074 seconds
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:13:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:14:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:14:00 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:14:00 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:14:00 [DEBUG   ] Finished updating board flags in 2.060 seconds
2018-11-19 20:14:19 [DEBUG   ] Got command BAM from MCS: ref #561879
2018-11-19 20:14:19 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:14:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561879  75 58441 72859021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:14:19 [DEBUG   ] Got command BAM from MCS: ref #561880
2018-11-19 20:14:19 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:14:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561880  75 58441 72859022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:19 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:39 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:14:58 [DEBUG   ] Got command BAM from MCS: ref #561881
2018-11-19 20:14:58 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:14:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561881  75 58441 72898025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:14:58 [DEBUG   ] Got command BAM from MCS: ref #561882
2018-11-19 20:14:58 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:14:58 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561882  75 58441 72898026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:14:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:14:59 [DEBUG   ] Got command RPT from MCS: ref #561883
2018-11-19 20:14:59 [DEBUG   ] summary = ERROR
2018-11-19 20:14:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561883  13 58441 72899430 A  ERRORERROR'
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:14:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:14:59 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:14:59 [DEBUG   ] Got command RPT from MCS: ref #561884
2018-11-19 20:14:59 [DEBUG   ] info = NUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28
2018-11-19 20:14:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561884 131 58441 72899633 A  ERRORNUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28'
2018-11-19 20:14:59 [DEBUG   ] Got command RPT from MCS: ref #561885
2018-11-19 20:14:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:14:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561885  12 58441 72899837 A  ERROR'
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:15:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:15:00 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:15:00 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:15:00 [DEBUG   ] Finished updating board flags in 2.024 seconds
2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:19 [DEBUG   ] Finished updating ant stats in 0.070 seconds
2018-11-19 20:15:37 [DEBUG   ] Got command BAM from MCS: ref #561886
2018-11-19 20:15:37 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:15:37 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561886  75 58441 72937029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:15:37 [DEBUG   ] Got command BAM from MCS: ref #561887
2018-11-19 20:15:37 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:15:37 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561887  75 58441 72937030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:39 [DEBUG   ] Finished updating ant stats in 0.076 seconds
2018-11-19 20:15:46 [DEBUG   ] Got command RPT from MCS: ref #561888
2018-11-19 20:15:46 [DEBUG   ] summary = ERROR
2018-11-19 20:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561888  13 58441 72946497 A  ERRORERROR'
2018-11-19 20:15:46 [DEBUG   ] Got command RPT from MCS: ref #561889
2018-11-19 20:15:46 [DEBUG   ] info = NUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28
2018-11-19 20:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561889 131 58441 72946498 A  ERRORNUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28'
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:15:58 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:15:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:15:59 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:16:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:16:00 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:16:00 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:16:00 [DEBUG   ] Finished updating board flags in 1.985 seconds
2018-11-19 20:16:16 [DEBUG   ] Got command BAM from MCS: ref #561909
2018-11-19 20:16:16 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:16:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561909  75 58441 72976021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:16:16 [DEBUG   ] Got command BAM from MCS: ref #561910
2018-11-19 20:16:16 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:16:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561910  75 58441 72976022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:19 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:16:39 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:16:55 [DEBUG   ] Got command BAM from MCS: ref #561912
2018-11-19 20:16:55 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:16:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561912  75 58441 73015021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:16:55 [DEBUG   ] Got command BAM from MCS: ref #561911
2018-11-19 20:16:55 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:16:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561911  75 58441 73015022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:16:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:00 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:17:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:17:00 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:17:00 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:17:00 [DEBUG   ] Finished updating board flags in 1.848 seconds
2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:20 [DEBUG   ] Finished updating ant stats in 0.071 seconds
2018-11-19 20:17:34 [DEBUG   ] Got command BAM from MCS: ref #561914
2018-11-19 20:17:34 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:17:34 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561914  75 58441 73054021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:17:34 [DEBUG   ] Got command BAM from MCS: ref #561913
2018-11-19 20:17:34 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:17:34 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561913  75 58441 73054022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:17:40 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:17:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:00 [DEBUG   ] Finished updating ant stats in 0.090 seconds
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:18:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:18:00 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:18:00 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:18:00 [DEBUG   ] Finished updating board flags in 1.773 seconds
2018-11-19 20:18:13 [DEBUG   ] Got command BAM from MCS: ref #561916
2018-11-19 20:18:13 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:18:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561916  75 58441 73093025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:18:13 [DEBUG   ] Got command BAM from MCS: ref #561915
2018-11-19 20:18:13 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:18:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561915  75 58441 73093026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:20 [DEBUG   ] Finished updating ant stats in 0.070 seconds
2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:18:40 [DEBUG   ] Finished updating ant stats in 0.071 seconds
2018-11-19 20:18:52 [DEBUG   ] Got command BAM from MCS: ref #561917
2018-11-19 20:18:52 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:18:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561917  75 58441 73132025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:18:52 [DEBUG   ] Got command BAM from MCS: ref #561918
2018-11-19 20:18:52 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:18:52 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561918  75 58441 73132026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:18:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:19:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:00 [DEBUG   ] Finished updating ant stats in 0.089 seconds
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:19:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:19:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:19:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:19:01 [DEBUG   ] Finished updating board flags in 2.187 seconds
2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:20 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:19:31 [DEBUG   ] Got command BAM from MCS: ref #561923
2018-11-19 20:19:31 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:19:31 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561923  75 58441 73171029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:19:31 [DEBUG   ] Got command BAM from MCS: ref #561924
2018-11-19 20:19:31 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:19:31 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561924  75 58441 73171030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:19:40 [DEBUG   ] Finished updating ant stats in 0.071 seconds
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:19:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:19:59 [DEBUG   ] Got command RPT from MCS: ref #561927
2018-11-19 20:19:59 [DEBUG   ] summary = ERROR
2018-11-19 20:19:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561927  13 58441 73199513 A  ERRORERROR'
2018-11-19 20:19:59 [DEBUG   ] Got command RPT from MCS: ref #561928
2018-11-19 20:19:59 [DEBUG   ] info = NUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28
2018-11-19 20:19:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561928 131 58441 73199717 A  ERRORNUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28'
2018-11-19 20:19:59 [DEBUG   ] Got command RPT from MCS: ref #561929
2018-11-19 20:19:59 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:19:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561929  12 58441 73199921 A  ERROR'
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:20:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:00 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:20:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:20:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:20:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:20:01 [DEBUG   ] Finished updating board flags in 2.002 seconds
2018-11-19 20:20:10 [DEBUG   ] Got command BAM from MCS: ref #561936
2018-11-19 20:20:10 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:20:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561936  75 58441 73210029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:20:10 [DEBUG   ] Got command BAM from MCS: ref #561935
2018-11-19 20:20:10 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:20:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561935  75 58441 73210030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:20 [DEBUG   ] Finished updating ant stats in 0.070 seconds
2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:20:40 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:20:46 [DEBUG   ] Got command RPT from MCS: ref #561937
2018-11-19 20:20:46 [DEBUG   ] summary = ERROR
2018-11-19 20:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561937  13 58441 73246501 A  ERRORERROR'
2018-11-19 20:20:46 [DEBUG   ] Got command RPT from MCS: ref #561938
2018-11-19 20:20:46 [DEBUG   ] info = NUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28
2018-11-19 20:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561938 131 58441 73246502 A  ERRORNUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28'
2018-11-19 20:20:49 [DEBUG   ] Got command BAM from MCS: ref #561955
2018-11-19 20:20:49 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:20:49 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561955  75 58441 73249021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:20:49 [DEBUG   ] Got command BAM from MCS: ref #561954
2018-11-19 20:20:49 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:20:49 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561954  75 58441 73249022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:20:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:21:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:21:00 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:21:00 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:21:00 [DEBUG   ] Finished updating board flags in 1.418 seconds
2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:00 [DEBUG   ] Finished updating ant stats in 0.068 seconds
2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:20 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:21:28 [DEBUG   ] Got command BAM from MCS: ref #561956
2018-11-19 20:21:28 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:21:28 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561956  75 58441 73288025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:21:28 [DEBUG   ] Got command BAM from MCS: ref #561957
2018-11-19 20:21:28 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:21:28 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561957  75 58441 73288026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:21:41 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:21:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:22:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:22:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:22:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:22:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:22:01 [DEBUG   ] Finished updating board flags in 1.983 seconds
2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:01 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:22:07 [DEBUG   ] Got command BAM from MCS: ref #561961
2018-11-19 20:22:07 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:22:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561961  75 58441 73327025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:22:07 [DEBUG   ] Got command BAM from MCS: ref #561960
2018-11-19 20:22:07 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:22:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561960  75 58441 73327026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:21 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:22:41 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:22:46 [DEBUG   ] Got command BAM from MCS: ref #561964
2018-11-19 20:22:46 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:22:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561964  75 58441 73366025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:22:46 [DEBUG   ] Got command BAM from MCS: ref #561965
2018-11-19 20:22:46 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:22:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561965  75 58441 73366026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:22:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:23:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:23:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:23:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:23:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:23:01 [DEBUG   ] Finished updating board flags in 2.022 seconds
2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:01 [DEBUG   ] Finished updating ant stats in 0.071 seconds
2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:21 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:23:25 [DEBUG   ] Got command BAM from MCS: ref #561966
2018-11-19 20:23:25 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:23:25 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561966  75 58441 73405021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:23:25 [DEBUG   ] Got command BAM from MCS: ref #561967
2018-11-19 20:23:25 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:23:25 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561967  75 58441 73405022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:23:41 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:23:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:24:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:24:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:24:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:24:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:24:01 [DEBUG   ] Finished updating board flags in 1.950 seconds
2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:01 [DEBUG   ] Finished updating ant stats in 0.070 seconds
2018-11-19 20:24:04 [DEBUG   ] Got command BAM from MCS: ref #561968
2018-11-19 20:24:04 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:24:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561968  75 58441 73444029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:24:04 [DEBUG   ] Got command BAM from MCS: ref #561969
2018-11-19 20:24:04 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:24:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561969  75 58441 73444030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:21 [DEBUG   ] Finished updating ant stats in 0.070 seconds
2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:24:41 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:24:43 [DEBUG   ] Got command BAM from MCS: ref #561970
2018-11-19 20:24:43 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:24:43 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561970  75 58441 73483021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:24:43 [DEBUG   ] Got command BAM from MCS: ref #561971
2018-11-19 20:24:43 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:24:43 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561971  75 58441 73483022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:24:59 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:24:59 [DEBUG   ] Got command RPT from MCS: ref #561972
2018-11-19 20:24:59 [DEBUG   ] summary = ERROR
2018-11-19 20:24:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561972  13 58441 73499593 A  ERRORERROR'
2018-11-19 20:24:59 [DEBUG   ] Got command RPT from MCS: ref #561973
2018-11-19 20:24:59 [DEBUG   ] info = NUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28
2018-11-19 20:24:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561973 131 58441 73499797 A  ERRORNUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28'
2018-11-19 20:25:00 [DEBUG   ] Got command RPT from MCS: ref #561974
2018-11-19 20:25:00 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:25:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561974  12 58441 73500001 A  ERROR'
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:25:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:25:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:25:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:25:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:25:01 [DEBUG   ] Finished updating board flags in 1.943 seconds
2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:01 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:21 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:25:22 [DEBUG   ] Got command BAM from MCS: ref #561979
2018-11-19 20:25:22 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:25:22 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561979  75 58441 73522021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:25:22 [DEBUG   ] Got command BAM from MCS: ref #561980
2018-11-19 20:25:22 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:25:22 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561980  75 58441 73522022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:25:41 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:25:46 [DEBUG   ] Got command RPT from MCS: ref #561981
2018-11-19 20:25:46 [DEBUG   ] summary = ERROR
2018-11-19 20:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561981  13 58441 73546493 A  ERRORERROR'
2018-11-19 20:25:46 [DEBUG   ] Got command RPT from MCS: ref #561982
2018-11-19 20:25:46 [DEBUG   ] info = NUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28
2018-11-19 20:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   561982 131 58441 73546494 A  ERRORNUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28'
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:26:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:26:01 [DEBUG   ] Got command BAM from MCS: ref #561998
2018-11-19 20:26:01 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:26:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561998  75 58441 73561029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:26:01 [DEBUG   ] Got command BAM from MCS: ref #561999
2018-11-19 20:26:01 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:26:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   561999  75 58441 73561030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:26:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:26:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:26:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:26:01 [DEBUG   ] Finished updating board flags in 1.934 seconds
2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:01 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:26:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:21 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:22 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:26:40 [DEBUG   ] Got command BAM from MCS: ref #562000
2018-11-19 20:26:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:26:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562000  75 58441 73600025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:26:40 [DEBUG   ] Got command BAM from MCS: ref #562001
2018-11-19 20:26:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:26:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562001  75 58441 73600026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:26:42 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:27:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:27:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:27:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:27:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:27:01 [DEBUG   ] Finished updating board flags in 1.752 seconds
2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:02 [DEBUG   ] Finished updating ant stats in 0.068 seconds
2018-11-19 20:27:19 [DEBUG   ] Got command BAM from MCS: ref #562003
2018-11-19 20:27:19 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:27:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562003  75 58441 73639025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:27:19 [DEBUG   ] Got command BAM from MCS: ref #562002
2018-11-19 20:27:19 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 20:27:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562002  75 58441 73639026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:22 [DEBUG   ] Finished updating ant stats in 0.075 seconds
2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:27:42 [DEBUG   ] Finished updating ant stats in 0.072 seconds
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:28:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:28:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:28:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:28:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:28:01 [DEBUG   ] Finished updating board flags in 1.657 seconds
2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:02 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:22 [DEBUG   ] Finished updating ant stats in 0.073 seconds
2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:28:42 [DEBUG   ] Finished updating ant stats in 0.071 seconds
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:29:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:29:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:29:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:29:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:29:01 [DEBUG   ] Finished updating board flags in 1.542 seconds
2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:02 [DEBUG   ] Finished updating ant stats in 0.068 seconds
2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:22 [DEBUG   ] Finished updating ant stats in 0.070 seconds
2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:29:42 [DEBUG   ] Finished updating ant stats in 0.070 seconds
2018-11-19 20:29:59 [DEBUG   ] Got command RPT from MCS: ref #562062
2018-11-19 20:29:59 [DEBUG   ] summary = ERROR
2018-11-19 20:29:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562062  13 58441 73799681 A  ERRORERROR'
2018-11-19 20:29:59 [DEBUG   ] Got command RPT from MCS: ref #562063
2018-11-19 20:29:59 [DEBUG   ] info = NUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28
2018-11-19 20:29:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562063 131 58441 73799885 A  ERRORNUM_BOARDS! 0x05! Boards missing due to communication problems - Found only 0 boards due to communication problems, need 28'
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa16, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa16, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa27, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa27, chip 5
2018-11-19 20:30:00 [DEBUG   ] Got command RPT from MCS: ref #562064
2018-11-19 20:30:00 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:30:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562064  12 58441 73800089 A  ERROR'
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa24, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa24, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa23, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa23, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa18, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa10, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa10, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa32, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa11, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa30, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa30, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa12, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa12, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa8, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa8, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa13, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa26, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa26, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 1
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 2
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 3
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 4
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-19 20:30:00 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa14, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa34, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa34, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa22, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa22, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa20, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa20, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa1, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa1, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa5, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa5, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa25, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa25, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa35, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa35, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa19, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa21, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa6, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa6, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa7, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa7, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa33, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa33, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa15, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa15, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 1
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 2
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 3
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 4
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCINT 0.000 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 20:30:01 [WARNING ] MonitorFlags: VCCAUX 0.000 out of range 2.400 to 2.60 on hostaname lwa28, chip 5
2018-11-19 20:30:01 [CRITICAL] Found only 0 boards due to unprogrammed FPGAs, need 28
2018-11-19 20:30:01 [CRITICAL] Found only 0 boards due to communication problems, need 28
2018-11-19 20:30:01 [DEBUG   ] Finished updating board flags in 1.460 seconds
2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [WARNING ] AntennaStats: command returned 1; error: FPGAs not programmed

2018-11-19 20:30:02 [DEBUG   ] Finished updating ant stats in 0.069 seconds
2018-11-19 20:30:17 [DEBUG   ] Got command INI from MCS: ref #562070
2018-11-19 20:30:17 [INFO    ] Parsing config file '/lwa/software/defaults.cfg'
2018-11-19 20:30:17 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_INI   562070   8 58441 73817461 ABOOTING'
2018-11-19 20:30:23 [DEBUG   ] Got command RPT from MCS: ref #562071
2018-11-19 20:30:23 [DEBUG   ] info = Running INI sequence
2018-11-19 20:30:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562071  28 58441 73823273 ABOOTINGRunning INI sequence'
2018-11-19 20:30:38 [INFO    ] -------------------
2018-11-19 20:30:38 [INFO    ]  LWA board status
2018-11-19 20:30:38 [INFO    ] -------------------
2018-11-19 20:30:38 [INFO    ] DP 1: 2 lwa27
2018-11-19 20:30:38 [INFO    ] DP 1: 3 lwa24
2018-11-19 20:30:38 [INFO    ] DP 1: 4 lwa23
2018-11-19 20:30:38 [INFO    ] DP 1: 5 lwa18
2018-11-19 20:30:38 [INFO    ] DP 1: 6 lwa10
2018-11-19 20:30:38 [INFO    ] DP 1: 7 lwa32
2018-11-19 20:30:38 [INFO    ] DP 1: 8 lwa11
2018-11-19 20:30:38 [INFO    ] DP 1: 9 lwa30
2018-11-19 20:30:38 [INFO    ] DP 1: 10 lwa12
2018-11-19 20:30:38 [INFO    ] DP 1: 11 lwa8
2018-11-19 20:30:38 [INFO    ] DP 1: 12 lwa13
2018-11-19 20:30:38 [INFO    ] DP 1: 13 lwa26
2018-11-19 20:30:38 [INFO    ] DP 1: 14 lwa14
2018-11-19 20:30:38 [INFO    ] DP 1: 16 lwa22
2018-11-19 20:30:38 [INFO    ] DP 1: 17 lwa20
2018-11-19 20:30:38 [INFO    ] DP 1: 18 lwa1
2018-11-19 20:30:38 [INFO    ] DP 1: 19 lwa5
2018-11-19 20:30:38 [INFO    ] DP 1: 20 lwa25
2018-11-19 20:30:38 [INFO    ] DP 1: 21 lwa35
2018-11-19 20:30:38 [INFO    ] DP 1: 22 lwa19
2018-11-19 20:30:38 [INFO    ] DP 1: 23 lwa21
2018-11-19 20:30:38 [INFO    ] DP 1: 24 lwa6
2018-11-19 20:30:38 [INFO    ] DP 1: 25 lwa7
2018-11-19 20:30:38 [INFO    ] DP 1: 26 lwa33
2018-11-19 20:30:38 [INFO    ] DP 1: 27 lwa15
2018-11-19 20:30:38 [INFO    ] DP 1: 28 lwa28
2018-11-19 20:30:38 [INFO    ] DP 2: 1 lwa16
2018-11-19 20:30:38 [INFO    ] DP 2: 15 lwa34
2018-11-19 20:30:38 [INFO    ] Total of 28 boards
2018-11-19 20:30:38 [INFO    ] Total of 260 stands
2018-11-19 20:30:38 [INFO    ] -------------------
2018-11-19 20:30:38 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:39 [DEBUG   ] sendHostnameCommand - Sent hostname lwa27 command /usr/local/bin/lwa27 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:39 [DEBUG   ] sendHostnameCommand - Sent hostname lwa24 command /usr/local/bin/lwa24 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:39 [DEBUG   ] sendHostnameCommand - Sent hostname lwa23 command /usr/local/bin/lwa23 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:39 [DEBUG   ] sendHostnameCommand - Sent hostname lwa18 command /usr/local/bin/lwa18 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:40 [DEBUG   ] sendHostnameCommand - Sent hostname lwa10 command /usr/local/bin/lwa10 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:40 [DEBUG   ] sendHostnameCommand - Sent hostname lwa32 command /usr/local/bin/lwa32 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:40 [DEBUG   ] sendHostnameCommand - Sent hostname lwa11 command /usr/local/bin/lwa11 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:40 [DEBUG   ] sendHostnameCommand - Sent hostname lwa30 command /usr/local/bin/lwa30 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:40 [DEBUG   ] sendHostnameCommand - Sent hostname lwa12 command /usr/local/bin/lwa12 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:41 [DEBUG   ] sendHostnameCommand - Sent hostname lwa8 command /usr/local/bin/lwa8 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:41 [DEBUG   ] sendHostnameCommand - Sent hostname lwa13 command /usr/local/bin/lwa13 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:41 [DEBUG   ] sendHostnameCommand - Sent hostname lwa26 command /usr/local/bin/lwa26 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:41 [DEBUG   ] sendHostnameCommand - Sent hostname lwa14 command /usr/local/bin/lwa14 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:42 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:42 [DEBUG   ] sendHostnameCommand - Sent hostname lwa22 command /usr/local/bin/lwa22 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:42 [DEBUG   ] sendHostnameCommand - Sent hostname lwa20 command /usr/local/bin/lwa20 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:42 [DEBUG   ] sendHostnameCommand - Sent hostname lwa1 command /usr/local/bin/lwa1 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:43 [DEBUG   ] sendHostnameCommand - Sent hostname lwa5 command /usr/local/bin/lwa5 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:43 [DEBUG   ] sendHostnameCommand - Sent hostname lwa25 command /usr/local/bin/lwa25 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:43 [DEBUG   ] sendHostnameCommand - Sent hostname lwa35 command /usr/local/bin/lwa35 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:43 [DEBUG   ] sendHostnameCommand - Sent hostname lwa19 command /usr/local/bin/lwa19 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:44 [DEBUG   ] sendHostnameCommand - Sent hostname lwa21 command /usr/local/bin/lwa21 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:44 [DEBUG   ] sendHostnameCommand - Sent hostname lwa6 command /usr/local/bin/lwa6 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:44 [DEBUG   ] sendHostnameCommand - Sent hostname lwa7 command /usr/local/bin/lwa7 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:44 [DEBUG   ] sendHostnameCommand - Sent hostname lwa33 command /usr/local/bin/lwa33 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:45 [DEBUG   ] sendHostnameCommand - Sent hostname lwa15 command /usr/local/bin/lwa15 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:45 [DEBUG   ] sendHostnameCommand - Sent hostname lwa28 command /usr/local/bin/lwa28 /lwa/software/dp_shutdown.sh
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa27', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14607
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa24', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14608
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa23', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14609
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa18', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14610
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa10', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14611
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa32', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14612
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa11', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14613
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa30', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14614
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa12', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14615
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa8', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14616
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa13', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14617
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa26', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14618
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa14', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14619
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa22', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14620
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa20', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14621
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa1', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14622
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa5', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14623
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa25', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14624
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa35', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14625
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa19', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14626
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa21', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14627
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa6', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14628
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa7', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14629
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa33', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14630
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa15', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14631
2018-11-19 20:30:45 [DEBUG   ] 2018-11-19 20:30:45 Starting ['/usr/local/bin/lwa28', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 14632
2018-11-19 20:30:46 [DEBUG   ] Got command RPT from MCS: ref #562072
2018-11-19 20:30:46 [DEBUG   ] summary = BOOTING
2018-11-19 20:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562072  15 58441 73846493 ABOOTINGBOOTING'
2018-11-19 20:30:46 [DEBUG   ] Got command RPT from MCS: ref #562073
2018-11-19 20:30:46 [DEBUG   ] info = Running INI sequence
2018-11-19 20:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562073  28 58441 73846494 ABOOTINGRunning INI sequence'
2018-11-19 20:30:57 [DEBUG   ] 2018-11-19 20:30:57 ['/usr/local/bin/lwa27', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14607 exited with status 0
2018-11-19 20:30:57 [DEBUG   ] 2018-11-19 20:30:57 ['/usr/local/bin/lwa24', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14608 exited with status 0
2018-11-19 20:30:57 [DEBUG   ] 2018-11-19 20:30:57 ['/usr/local/bin/lwa23', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14609 exited with status 0
2018-11-19 20:30:57 [DEBUG   ] 2018-11-19 20:30:57 ['/usr/local/bin/lwa18', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14610 exited with status 0
2018-11-19 20:30:57 [DEBUG   ] 2018-11-19 20:30:57 ['/usr/local/bin/lwa10', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14611 exited with status 0
2018-11-19 20:30:57 [DEBUG   ] 2018-11-19 20:30:57 ['/usr/local/bin/lwa32', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14612 exited with status 0
2018-11-19 20:30:57 [DEBUG   ] 2018-11-19 20:30:57 ['/usr/local/bin/lwa11', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14613 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa30', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14614 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa12', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14615 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa8', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14616 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa13', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14617 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa26', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14618 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa14', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14619 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa22', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14620 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa20', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14621 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa1', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14622 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa5', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14623 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa25', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14624 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa35', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14625 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa19', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14626 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa21', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14627 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa6', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14628 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa7', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14629 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa33', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14630 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa15', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14631 exited with status 0
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 ['/usr/local/bin/lwa28', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 14632 exited with status 0
2018-11-19 20:30:58 [INFO    ] Finish loading DP1 firmware'/lwa/hardware/lwa_dp_top_2_1_1_49.bin'
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa27', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa24', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa23', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa18', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa10', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa32', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa11', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa30', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa12', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa8', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa13', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa26', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa14', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa22', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa20', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa1', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa5', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa25', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa35', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa19', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa21', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa6', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa7', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa33', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa15', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] spawn_process: at 2018-11-19 20:30:58 running ['/usr/local/bin/lwa28', '/lwa/software/dp1_init.sh']
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 Starting ['/usr/local/bin/lwa16', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] pid 14659
2018-11-19 20:30:58 [DEBUG   ] 2018-11-19 20:30:58 Starting ['/usr/local/bin/lwa34', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] pid 14660
2018-11-19 20:31:03 [DEBUG   ] 2018-11-19 20:31:03 ['/usr/local/bin/lwa16', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] Process 14659 exited with status 0
2018-11-19 20:31:03 [DEBUG   ] 2018-11-19 20:31:03 ['/usr/local/bin/lwa34', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] Process 14660 exited with status 0
2018-11-19 20:31:03 [INFO    ] Finish loading DP2 firmware '/lwa/hardware/lwa_drx_top_3_2_0_6.bin'
2018-11-19 20:31:03 [DEBUG   ] spawn_process: at 2018-11-19 20:31:03 running ['/usr/local/bin/lwa16', '/lwa/software/dp2_init.sh']
2018-11-19 20:31:03 [DEBUG   ] spawn_process: at 2018-11-19 20:31:03 running ['/usr/local/bin/lwa34', '/lwa/software/dp2_init.sh']
2018-11-19 20:31:03 [INFO    ] Setting Beam 1 xbars
2018-11-19 20:31:03 [DEBUG   ] board 11 xbars (bd11 to bd5 through bot5) - beam 1
2018-11-19 20:31:04 [DEBUG   ] board 5 xbars (bd5 from bd11 through bot1) - beam 1
2018-11-19 20:31:04 [DEBUG   ] board 5 xbars (bd5 to bd13 through bot5) - beam 1
2018-11-19 20:31:04 [DEBUG   ] board 13 xbars (bd13 from bd5 through bot1) - beam 1
2018-11-19 20:31:04 [DEBUG   ] board 13 xbars (bd13 to bd7 through bot5) - beam 1
2018-11-19 20:31:04 [DEBUG   ] board 7 xbars (bd7 from bd13 through bot1) - beam 1
2018-11-19 20:31:04 [DEBUG   ] board 7 xbars (bd7 to bd9 through bot5) - beam 1
2018-11-19 20:31:04 [DEBUG   ] board 9 xbars (bd9 from bd7 through bot1) - beam 1
2018-11-19 20:31:04 [DEBUG   ] board 9 xbars (bd9 to bd3 through bot5) - beam 1
2018-11-19 20:31:04 [DEBUG   ] board 3 xbars (bd3 from bd9 through bot1) - beam 1
2018-11-19 20:31:05 [DEBUG   ] board 3 xbars (bd3 to bd2 through bot5) - beam 1
2018-11-19 20:31:05 [DEBUG   ] board 2 xbars (bd2 from bd3 through bot1) - beam 1
2018-11-19 20:31:05 [DEBUG   ] board 2 xbars (bd2 to bd6 through bot5) - beam 1
2018-11-19 20:31:05 [DEBUG   ] board 6 xbars (bd6 from bd2 through bot1) - beam 1
2018-11-19 20:31:05 [DEBUG   ] board 6 xbars (bd6 to bd12 through bot5) - beam 1
2018-11-19 20:31:05 [DEBUG   ] board 12 xbars (bd12 from bd6 through bot1) - beam 1
2018-11-19 20:31:05 [DEBUG   ] board 12 xbars (bd12 to bd8 through bot5) - beam 1
2018-11-19 20:31:05 [DEBUG   ] board 8 xbars (bd8 from bd12 through bot1) - beam 1
2018-11-19 20:31:05 [DEBUG   ] board 8 xbars (bd8 to bd4 through bot5) - beam 1
2018-11-19 20:31:06 [DEBUG   ] board 4 xbars (bd4 from bd8 through bot1) - beam 1
2018-11-19 20:31:06 [DEBUG   ] board 4 xbars (bd4 to bd10 through bot5) - beam 1
2018-11-19 20:31:06 [DEBUG   ] board 10 xbars (bd10 from bd4 through bot1) - beam 1
2018-11-19 20:31:06 [DEBUG   ] board 10 xbars (bd10 to bd14 through bot5) - beam 1
2018-11-19 20:31:06 [DEBUG   ] board 14 xbars (bd14 from bd10 through top1) - beam 1
2018-11-19 20:31:06 [DEBUG   ] board 28 xbars  (bd 28 to bd24 through top1)- beam 1
2018-11-19 20:31:06 [DEBUG   ] board 24 xbars (bd24 from bd28 through bot5) - beam 1
2018-11-19 20:31:06 [DEBUG   ] board 24 xbars (bd24 to bd18 through bot1) - beam 1
2018-11-19 20:31:06 [DEBUG   ] board 18 xbars (bd18 from bd24 through bot5) - beam 1
2018-11-19 20:31:06 [DEBUG   ] Got command RPT from MCS: ref #562093
2018-11-19 20:31:06 [DEBUG   ] info = Starting calibration
2018-11-19 20:31:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562093  28 58441 73866978 ABOOTINGStarting calibration'
2018-11-19 20:31:07 [DEBUG   ] board 18 xbars (bd18 to bd22 through bot1) - beam 1
2018-11-19 20:31:07 [DEBUG   ] board 22 xbars (bd22 from bd18 through bot5) - beam 1
2018-11-19 20:31:07 [DEBUG   ] board 22 xbars (bd22 to bd26 through bot1) - beam 1
2018-11-19 20:31:07 [DEBUG   ] board 26 xbars (bd26 from bd22 through bot5) - beam 1
2018-11-19 20:31:07 [DEBUG   ] board 26 xbars (bd26 to bd20 through bot1) - beam 1
2018-11-19 20:31:07 [DEBUG   ] board 20 xbars (bd20 from bd26 through bot5) - beam 1
2018-11-19 20:31:07 [DEBUG   ] board 20 xbars (bd20 to bd16 through bot1) - beam 1
2018-11-19 20:31:07 [DEBUG   ] board 16 xbars (bd16 from bd20 through bot5) - beam 1
2018-11-19 20:31:07 [DEBUG   ] board 16 xbars (bd16 to bd17 through bot1) - beam 1
2018-11-19 20:31:07 [DEBUG   ] board 17 xbars (bd17 from bd16 through bot5) - beam 1
2018-11-19 20:31:08 [DEBUG   ] board 17 xbars (bd17 to bd23 through bot1) - beam 1
2018-11-19 20:31:08 [DEBUG   ] board 23 xbars (bd23 from bd17 through bot5) - beam 1
2018-11-19 20:31:08 [DEBUG   ] board 23 xbars (bd23 to bd21 through bot1) - beam 1
2018-11-19 20:31:08 [DEBUG   ] board 21 xbars (bd21 from bd23 through bot5) - beam 1
2018-11-19 20:31:08 [DEBUG   ] board 21 xbars (bd21 to bd27 through bot1) - beam 1
2018-11-19 20:31:08 [DEBUG   ] board 27 xbars (bd27 from bd21 through bot5) - beam 1
2018-11-19 20:31:08 [DEBUG   ] board 27 xbars (bd27 to bd19 through bot1) - beam 1
2018-11-19 20:31:08 [DEBUG   ] board 19 xbars (bd19 from bd27 through bot5) - beam 1
2018-11-19 20:31:08 [DEBUG   ] board 19 xbars (bd19 to bd25 through bot1) - beam 1
2018-11-19 20:31:09 [DEBUG   ] board 25 xbars (bd25 from bd19 through bot5) - beam 1
2018-11-19 20:31:09 [DEBUG   ] board 25 xbars (bd25 to bd15 through bot1) - beam 1
2018-11-19 20:31:09 [DEBUG   ] board 15 xbars (board 15 from board 25 through bot5) - beam 1
2018-11-19 20:31:09 [INFO    ] Setting Beam 2 xbars
2018-11-19 20:31:09 [DEBUG   ] board 3 xbars (bd3 to bd7 through top1) - beam 2
2018-11-19 20:31:09 [DEBUG   ] board 7 xbars (bd7 from bd3 through top5) - beam 2
2018-11-19 20:31:09 [DEBUG   ] board 7 xbars (bd7 to bd11 through top1) - beam 2
2018-11-19 20:31:09 [DEBUG   ] board 11 xbars (bd11 from bd7 through top5) - beam 2
2018-11-19 20:31:09 [DEBUG   ] board 11 xbars (bd11 to bd13 through top1) - beam 2
2018-11-19 20:31:09 [DEBUG   ] board 13 xbars (bd13 from bd11 through top5) - beam 2
2018-11-19 20:31:10 [DEBUG   ] board 13 xbars (bd13 to bd9 through top1) - beam 2
2018-11-19 20:31:10 [DEBUG   ] board 9 xbars (bd9 from bd13 through top5) - beam 2
2018-11-19 20:31:10 [DEBUG   ] board 9 xbars (bd9 to bd5 through top1) - beam 2
2018-11-19 20:31:10 [DEBUG   ] board 5 xbars (bd5 from bd9 through top5) - beam 2
2018-11-19 20:31:10 [DEBUG   ] board 5 xbars (bd5 to bd2 through top1) - beam 2
2018-11-19 20:31:10 [DEBUG   ] board 2 xbars (bd2 from bd5 through top5) - beam 2
2018-11-19 20:31:10 [DEBUG   ] board 2 xbars (bd2 to bd4 through top1) - beam 2
2018-11-19 20:31:10 [DEBUG   ] board 4 xbars (bd4 from bd2 through top5) - beam 2
2018-11-19 20:31:10 [DEBUG   ] board 4 xbars (bd4 to bd6 through top1) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 6 xbars (bd6 from bd4 through top5) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 6 xbars (bd6 to bd8 through top1) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 8 xbars (bd8 from bd6 through top5) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 8 xbars (bd8 to bd10 through top1) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 10 xbars (bd10 from bd8 through top5) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 10 xbars (bd10 to bd12 through top1) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 12 xbars (bd12 from bd10 through top5) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 12 xbars (bd12 to bd14 through top1) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 14 xbars (bd14 from bd12 through bot5) - beam 2
2018-11-19 20:31:11 [DEBUG   ] board 28 xbars (bd28 to bd26 through bot5) - beam 2
2018-11-19 20:31:12 [DEBUG   ] board 26 xbars (bd26 from bd28 through top1) - beam 2
2018-11-19 20:31:12 [DEBUG   ] board 26 xbars (bd26 to bd24 through top5) - beam 2
2018-11-19 20:31:12 [DEBUG   ] board 24 xbars (bd24 from bd26 through top1) - beam 2
2018-11-19 20:31:12 [DEBUG   ] board 24 xbars (bd24 to bd22 through top5) - beam 2
2018-11-19 20:31:12 [DEBUG   ] board 22 xbars (bd22 from bd24 through top1) - beam 2
2018-11-19 20:31:12 [DEBUG   ] board 22 xbars (bd22 to bd20 through top5) - beam 2
2018-11-19 20:31:12 [DEBUG   ] board 20 xbars (bd20 from bd22 through top1) - beam 2
2018-11-19 20:31:12 [DEBUG   ] board 20 xbars (bd20 to bd18 through top5) - beam 2
2018-11-19 20:31:12 [DEBUG   ] board 18 xbars (bd18 from bd20 through top1) - beam 2
2018-11-19 20:31:13 [DEBUG   ] board 18 xbars (bd18 to bd16 through top5) - beam 2
2018-11-19 20:31:13 [DEBUG   ] board 16 xbars (bd16 from bd18 through top1) - beam 2
2018-11-19 20:31:13 [DEBUG   ] board 16 xbars (bd16 to bd19 through top5) - beam 2
2018-11-19 20:31:13 [DEBUG   ] board 19 xbars (bd19 from bd16 through top1) - beam 2
2018-11-19 20:31:13 [DEBUG   ] board 19 xbars (bd19 to bd23 through top5) - beam 2
2018-11-19 20:31:13 [DEBUG   ] board 23 xbars (bd23 from bd19 through top1) - beam 2
2018-11-19 20:31:13 [DEBUG   ] board 23 xbars (bd23 to bd27 through top5) - beam 2
2018-11-19 20:31:13 [DEBUG   ] board 27 xbars (bd27 from bd23 through top1) - beam 2
2018-11-19 20:31:13 [DEBUG   ] board 27 xbars (bd27 to bd25 through top5) - beam 2
2018-11-19 20:31:14 [DEBUG   ] board 25 xbars (bd25 from bd27 through top1) - beam 2
2018-11-19 20:31:14 [DEBUG   ] board 25 xbars (bd25 to bd21 through top5) - beam 2
2018-11-19 20:31:14 [DEBUG   ] board 21 xbars (bd21 from bd25 through top1) - beam 2
2018-11-19 20:31:14 [DEBUG   ] board 21 xbars (bd21 to bd17 through top5) - beam 2
2018-11-19 20:31:14 [DEBUG   ] board 17 xbars (bd17 from bd21 through top1) - beam 2
2018-11-19 20:31:14 [DEBUG   ] board 17 xbars (bd17 to bd15 through top5) - beam 2
2018-11-19 20:31:14 [DEBUG   ] board 15 xbars (bd15 from bd17 through top1) - beam 2
2018-11-19 20:31:14 [INFO    ] Setting Beam 3 xbars
2018-11-19 20:31:14 [DEBUG   ] board 25 xbars (bd25 to bd19 through bot5) - beam 3
2018-11-19 20:31:14 [DEBUG   ] board 19 xbars (bd19 from bd25 through bot1) - beam 3
2018-11-19 20:31:15 [DEBUG   ] board 19 xbars (bd19 to bd27 through bot5) - beam 3
2018-11-19 20:31:15 [DEBUG   ] board 27 xbars (bd27 from bd19 through bot1) - beam 3
2018-11-19 20:31:15 [DEBUG   ] board 27 xbars (bd27 to bd21 through bot5) - beam 3
2018-11-19 20:31:15 [DEBUG   ] board 21 xbars (bd21 from bd27 through bot1) - beam 3
2018-11-19 20:31:15 [DEBUG   ] board 21 xbars (bd21 to bd23 through bot5) - beam 3
2018-11-19 20:31:15 [DEBUG   ] board 23 xbars (bd23 from bd21 through bot1) - beam 3
2018-11-19 20:31:15 [DEBUG   ] board 23 xbars (bd23 to bd17 through bot5) - beam 3
2018-11-19 20:31:15 [DEBUG   ] board 17 xbars (bd17 from bd23 through bot1) - beam 3
2018-11-19 20:31:15 [DEBUG   ] Got command RPT from MCS: ref #562094
2018-11-19 20:31:15 [DEBUG   ] info = Starting calibration
2018-11-19 20:31:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562094  28 58441 73875785 ABOOTINGStarting calibration'
2018-11-19 20:31:15 [DEBUG   ] board 17 xbars (bd17 to bd16 through bot5) - beam 3
2018-11-19 20:31:15 [DEBUG   ] board 16 xbars (bd16 from bd17 through bot1) - beam 3
2018-11-19 20:31:16 [DEBUG   ] board 16 xbars (bd16 to bd20 through bot5) - beam 3
2018-11-19 20:31:16 [DEBUG   ] board 20 xbars (bd20 from bd16 through bot1) - beam 3
2018-11-19 20:31:16 [DEBUG   ] board 20 xbars (bd20 to bd26 through bot5) - beam 3
2018-11-19 20:31:16 [DEBUG   ] board 26 xbars (bd26 from bd20 through bot1) - beam 3
2018-11-19 20:31:16 [DEBUG   ] board 26 xbars (bd26 to bd22 through bot5) - beam 3
2018-11-19 20:31:16 [DEBUG   ] board 22 xbars (bd22 from bd26 through bot1) - beam 3
2018-11-19 20:31:16 [DEBUG   ] board 22 xbars (bd22 to bd18 through bot5) - beam 3
2018-11-19 20:31:16 [DEBUG   ] board 18 xbars (bd18 from bd22 through bot1) - beam 3
2018-11-19 20:31:16 [DEBUG   ] board 18 xbars (bd18 to bd24 through bot5) - beam 3
2018-11-19 20:31:17 [DEBUG   ] board 24 xbars (bd24 from bd18 through bot1) - beam 3
2018-11-19 20:31:17 [DEBUG   ] board 24 xbars (bd24 to bd28 through bot5) - beam 3
2018-11-19 20:31:17 [DEBUG   ] board 28 xbars (bd28 from bd24 through top1) - beam 3
2018-11-19 20:31:17 [DEBUG   ] board 14 xbars (bd14 to bd10 through top1) - beam 3
2018-11-19 20:31:17 [DEBUG   ] board 10 xbars (bd10 from bd14 through bot5) - beam 3
2018-11-19 20:31:17 [DEBUG   ] board 10 xbars (bd10 to bd4 through bot1) - beam 3
2018-11-19 20:31:17 [DEBUG   ] board 4 xbars (bd4 from bd10 through bot5) - beam 3
2018-11-19 20:31:17 [DEBUG   ] board 4 xbars (bd4 to bd8 through bot1) - beam 3
2018-11-19 20:31:17 [DEBUG   ] board 8 xbars (bd8 from bd4 through bot5) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 8 xbars (bd8 to bd12 through bot1) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 12 xbars (bd12 from bd8 through bot5) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 12 xbars (bd12 to bd6 through bot1) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 6 xbars (bd6 from bd12 through bot5) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 6 xbars (bd6 to bd2 through bot1) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 2 xbars (bd2 from bd6 through bot5) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 2 xbars (bd2 to bd3 through bot1) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 3 xbars (bd3 from bd2 through bot5) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 3 xbars (bd3 to bd9 through bot1) - beam 3
2018-11-19 20:31:18 [DEBUG   ] board 9 xbars (bd9 from bd3 through bot5) - beam 3
2018-11-19 20:31:19 [DEBUG   ] board 9 xbars (bd9 to bd7 through bot1) - beam 3
2018-11-19 20:31:19 [DEBUG   ] board 7 xbars (bd7 from bd9 through bot5) - beam 3
2018-11-19 20:31:19 [DEBUG   ] board 7 xbars (bd7 to bd13 through bot1) - beam 3
2018-11-19 20:31:19 [DEBUG   ] board 13 xbars (bd13 from bd7 through bot5) - beam 3
2018-11-19 20:31:19 [DEBUG   ] board 13 xbars (bd13 to bd5 through bot1) - beam 3
2018-11-19 20:31:19 [DEBUG   ] board 5 xbars (bd5 from bd13 through bot5) - beam 3
2018-11-19 20:31:19 [DEBUG   ] board 5 xbars (bd5 to bd11 through bot1) - beam 3
2018-11-19 20:31:19 [DEBUG   ] board 11 xbars (bd11 from bd5 through bot5) - beam 3
2018-11-19 20:31:19 [DEBUG   ] board 11 xbars (bd11 to bd1 through bot1) - beam 3
2018-11-19 20:31:20 [DEBUG   ] board 1 xbars (bd1 from bd11 through bot5) - beam 3
2018-11-19 20:31:20 [INFO    ] Setting Beam 4 xbars
2018-11-19 20:31:20 [DEBUG   ] board 17 xbars (bd17 to bd21 through top1) - beam 4
2018-11-19 20:31:20 [DEBUG   ] board 21 xbars (bd21 from bd17 through top5) - beam 4
2018-11-19 20:31:20 [DEBUG   ] board 21 xbars (bd21 to bd25 through top1) - beam 4
2018-11-19 20:31:20 [DEBUG   ] board 21 xbars (bd21 from bd17 through top5) - beam 4
2018-11-19 20:31:20 [DEBUG   ] board 21 xbars (bd21 to bd25 through top1) - beam 4
2018-11-19 20:31:20 [DEBUG   ] board 25 xbars (bd25 from bd21 through top5) - beam 4
2018-11-19 20:31:20 [DEBUG   ] board 25 xbars (bd25 to bd27 through top1) - beam 4
2018-11-19 20:31:20 [DEBUG   ] board 27 xbars (bd27 from bd25 through top5) - beam 4
2018-11-19 20:31:21 [DEBUG   ] board 27 xbars (bd27 to bd23 through top1) - beam 4
2018-11-19 20:31:21 [DEBUG   ] board 23 xbars (bd23 from bd27 through top5) - beam 4
2018-11-19 20:31:21 [DEBUG   ] board 23 xbars (bd23 to bd19 through top1) - beam 4
2018-11-19 20:31:21 [DEBUG   ] board 19 xbars (bd19 from bd23 through top5) - beam 4
2018-11-19 20:31:21 [DEBUG   ] board 19 xbars (bd19 to bd16 through top1) - beam 4
2018-11-19 20:31:21 [DEBUG   ] board 16 xbars (bd16 from bd19 through top5) - beam 4
2018-11-19 20:31:21 [DEBUG   ] board 16 xbars (bd16 to bd18 through top1) - beam 4
2018-11-19 20:31:21 [DEBUG   ] board 18 xbars (bd18 from bd16 through top5) - beam 4
2018-11-19 20:31:21 [DEBUG   ] board 18 xbars (bd18 to bd20 through top1) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 20 xbars (bd20 from bd18 through top5) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 20 xbars (bd20 to bd22 through top1) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 22 xbars (bd22 from bd20 through top5) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 22 xbars (bd22 to bd24 through top1) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 24 xbars (bd24 from bd22 through top5) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 24 xbars (bd24 to bd26 through top1) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 26 xbars (bd26 from bd24 through top5) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 26 xbars (bd26 to bd28 through top1) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 28 xbars (bd28 from bd26 through bot5) - beam 4
2018-11-19 20:31:22 [DEBUG   ] board 14 xbars (bd14 to bd12 through bot5) - beam 4
2018-11-19 20:31:23 [DEBUG   ] board 12 xbars (bd12 from bd14 through top1) - beam 4
2018-11-19 20:31:23 [DEBUG   ] board 12 xbars (bd12 to bd10 through top5) - beam 4
2018-11-19 20:31:23 [DEBUG   ] board 10 xbars (bd10 from bd12 through top1) - beam 4
2018-11-19 20:31:23 [DEBUG   ] board 10 xbars (bd10 to bd8 through top5) - beam 4
2018-11-19 20:31:23 [DEBUG   ] board 8 xbars (bd8 from bd10 through top1) - beam 4
2018-11-19 20:31:23 [DEBUG   ] board 8 xbars (bd8 to bd6 through top5) - beam 4
2018-11-19 20:31:23 [DEBUG   ] board 6 xbars (bd6 from bd8 through top1) - beam 4
2018-11-19 20:31:23 [DEBUG   ] board 6 xbars (bd6 to bd4 through top5) - beam 4
2018-11-19 20:31:23 [DEBUG   ] board 4 xbars (bd4 from bd6 through top1) - beam 4
2018-11-19 20:31:24 [DEBUG   ] board 4 xbars (bd4 to bd2 through top5) - beam 4
2018-11-19 20:31:24 [DEBUG   ] board 2 xbars (bd2 from bd4 through top1) - beam 4
2018-11-19 20:31:24 [DEBUG   ] board 2 xbars (bd2 to bd5 through top5) - beam 4
2018-11-19 20:31:24 [DEBUG   ] board 5 xbars (bd5 from bd2 through top1) - beam 4
2018-11-19 20:31:24 [DEBUG   ] board 5 xbars (bd5 to bd9 through top5) - beam 4
2018-11-19 20:31:24 [DEBUG   ] board 9 xbars (bd9 from bd5 through top1) - beam 4
2018-11-19 20:31:24 [DEBUG   ] board 9 xbars (bd9 to bd13 through top5) - beam 4
2018-11-19 20:31:24 [DEBUG   ] board 13 xbars (bd13 from bd9 through top1) - beam 4
2018-11-19 20:31:24 [DEBUG   ] board 13 xbars (bd13 to bd11 through top5) - beam 4
2018-11-19 20:31:25 [DEBUG   ] board 11 xbars (bd11 from bd13 through top1) - beam 4
2018-11-19 20:31:25 [DEBUG   ] board 11 xbars (bd11 to bd7 through top5) - beam 4
2018-11-19 20:31:25 [DEBUG   ] board 7 xbars (bd7 from bd11 through top1) - beam 4
2018-11-19 20:31:25 [DEBUG   ] board 7 xbars (bd7 to bd3 through top5) - beam 4
2018-11-19 20:31:25 [DEBUG   ] board 3 xbars (bd3 from bd7 through top1) - beam 4
2018-11-19 20:31:25 [DEBUG   ] board 3 xbars (bd3 to bd1 through top5) - beam 4
2018-11-19 20:31:25 [DEBUG   ] board 1 xbars (bd1 from bd3 through top1) - beam 4
2018-11-19 20:31:25 [INFO    ] All xbars set
2018-11-19 20:31:25 [INFO    ] Setting adder chain routing for all beams
2018-11-19 20:31:30 [INFO    ] Setting beam ids
2018-11-19 20:31:40 [INFO    ] Starting beam delay calibration
2018-11-19 20:31:52 [INFO    ] Beam 1 latency is 6543
2018-11-19 20:31:52 [INFO    ] Beam 2 latency is 6593
2018-11-19 20:31:52 [INFO    ] Beam 3 latency is 358
2018-11-19 20:31:52 [INFO    ] Beam 4 latency is 6571
2018-11-19 20:31:52 [INFO    ] Stop calibration
2018-11-19 20:31:59 [DEBUG   ] Got command RPT from MCS: ref #562095
2018-11-19 20:31:59 [DEBUG   ] info = Starting calibration
2018-11-19 20:31:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562095  28 58441 73919013 ABOOTINGStarting calibration'
2018-11-19 20:32:00 [ERROR   ] Beam delay calibration failed
2018-11-19 20:32:00 [CRITICAL] Calibration failed
2018-11-19 20:32:00 [DEBUG   ] spawn_process: at 2018-11-19 20:32:00 running ['/usr/local/bin/lwa34', '/lwa/software/cfg_eth 4 1 192.168.0.1 192.168.0.33 10000']
2018-11-19 20:32:00 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 5 0x138 8909
2018-11-19 20:32:00 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 5 0x13c 9111
2018-11-19 20:32:00 [DEBUG   ] spawn_process: at 2018-11-19 20:32:00 running ['/usr/local/bin/lwa34', '/lwa/software/cfg_eth 2 1 192.168.0.2 192.168.0.34 10000']
2018-11-19 20:32:00 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 1 0x138 8a0a
2018-11-19 20:32:01 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 1 0x13c 9212
2018-11-19 20:32:01 [DEBUG   ] spawn_process: at 2018-11-19 20:32:01 running ['/usr/local/bin/lwa16', '/lwa/software/cfg_eth 4 1 192.168.0.3 192.168.0.35 10000']
2018-11-19 20:32:01 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 5 0x138 8b0b
2018-11-19 20:32:01 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 5 0x13c 9313
2018-11-19 20:32:01 [DEBUG   ] spawn_process: at 2018-11-19 20:32:01 running ['/usr/local/bin/lwa16', '/lwa/software/cfg_eth 2 1 192.168.0.4 192.168.0.36 10000']
2018-11-19 20:32:01 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 1 0x138 8c0c
2018-11-19 20:32:01 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 1 0x13c 9414
2018-11-19 20:32:01 [DEBUG   ] SetDRXTime: DP2 - DP/MCS NTP skew is -2 msec
2018-11-19 20:32:01 [DEBUG   ] SetDRXTime: started set_drx_time on hostname lwa16 at 302 ms (1542659521.304 s) with PID 16220 and time 1542659522
2018-11-19 20:32:01 [DEBUG   ] SetDRXTime: DP2 - DP/MCS NTP skew is -2 msec
2018-11-19 20:32:01 [DEBUG   ] SetDRXTime: started set_drx_time on hostname lwa34 at 390 ms (1542659521.392 s) with PID 16223 and time 1542659522
2018-11-19 20:32:02 [DEBUG   ] Finished updating ant stats in 0.860 seconds
2018-11-19 20:32:02 [INFO    ] Started antennas statistics background thread
2018-11-19 20:32:02 [INFO    ] Started board flags background thread
2018-11-19 20:32:02 [INFO    ] Started BAM background thread
2018-11-19 20:32:02 [INFO    ] Finished the INI process (ref. #562070) in 105.060 s
2018-11-19 20:32:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:32:03 [WARNING ] (13, 1, 'bot')
2018-11-19 20:32:03 [WARNING ] (13, 1, 'bot')
2018-11-19 20:32:03 [WARNING ] (13, 1, 'bot')
2018-11-19 20:32:03 [WARNING ] (13, 1, 'bot')
2018-11-19 20:32:03 [CRITICAL] MGT underflow detected on beam(s) 1, 2, 3, 4
2018-11-19 20:32:03 [DEBUG   ] Finished updating board flags in 1.031 seconds
2018-11-19 20:32:22 [DEBUG   ] Finished updating ant stats in 0.791 seconds
2018-11-19 20:32:25 [DEBUG   ] Got command RPT from MCS: ref #562096
2018-11-19 20:32:25 [DEBUG   ] info = T_NOM1 T_NOM2 T_NOM3 T_NOM4! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:32:25 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562096  98 58441 73945393 A  ERRORT_NOM1 T_NOM2 T_NOM3 T_NOM4! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:32:31 [DEBUG   ] Got command INI from MCS: ref #562097
2018-11-19 20:32:31 [INFO    ] Parsing config file '/lwa/software/defaults.cfg'
2018-11-19 20:32:31 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_INI   562097   8 58441 73951806 ABOOTING'
2018-11-19 20:32:52 [INFO    ] -------------------
2018-11-19 20:32:52 [INFO    ]  LWA board status
2018-11-19 20:32:52 [INFO    ] -------------------
2018-11-19 20:32:52 [INFO    ] DP 1: 2 lwa27
2018-11-19 20:32:52 [INFO    ] DP 1: 3 lwa24
2018-11-19 20:32:52 [INFO    ] DP 1: 4 lwa23
2018-11-19 20:32:52 [INFO    ] DP 1: 5 lwa18
2018-11-19 20:32:52 [INFO    ] DP 1: 6 lwa10
2018-11-19 20:32:52 [INFO    ] DP 1: 7 lwa32
2018-11-19 20:32:52 [INFO    ] DP 1: 8 lwa11
2018-11-19 20:32:52 [INFO    ] DP 1: 9 lwa30
2018-11-19 20:32:52 [INFO    ] DP 1: 10 lwa12
2018-11-19 20:32:52 [INFO    ] DP 1: 11 lwa8
2018-11-19 20:32:52 [INFO    ] DP 1: 12 lwa13
2018-11-19 20:32:52 [INFO    ] DP 1: 13 lwa26
2018-11-19 20:32:52 [INFO    ] DP 1: 14 lwa14
2018-11-19 20:32:52 [INFO    ] DP 1: 16 lwa22
2018-11-19 20:32:52 [INFO    ] DP 1: 17 lwa20
2018-11-19 20:32:52 [INFO    ] DP 1: 18 lwa1
2018-11-19 20:32:52 [INFO    ] DP 1: 19 lwa5
2018-11-19 20:32:52 [INFO    ] DP 1: 20 lwa25
2018-11-19 20:32:52 [INFO    ] DP 1: 21 lwa35
2018-11-19 20:32:52 [INFO    ] DP 1: 22 lwa19
2018-11-19 20:32:52 [INFO    ] DP 1: 23 lwa21
2018-11-19 20:32:52 [INFO    ] DP 1: 24 lwa6
2018-11-19 20:32:52 [INFO    ] DP 1: 25 lwa7
2018-11-19 20:32:52 [INFO    ] DP 1: 26 lwa33
2018-11-19 20:32:52 [INFO    ] DP 1: 27 lwa15
2018-11-19 20:32:52 [INFO    ] DP 1: 28 lwa28
2018-11-19 20:32:52 [INFO    ] DP 2: 1 lwa16
2018-11-19 20:32:52 [INFO    ] DP 2: 15 lwa34
2018-11-19 20:32:52 [INFO    ] Total of 28 boards
2018-11-19 20:32:52 [INFO    ] Total of 260 stands
2018-11-19 20:32:52 [INFO    ] -------------------
2018-11-19 20:32:52 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:53 [DEBUG   ] sendHostnameCommand - Sent hostname lwa27 command /usr/local/bin/lwa27 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:53 [DEBUG   ] sendHostnameCommand - Sent hostname lwa24 command /usr/local/bin/lwa24 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:53 [DEBUG   ] sendHostnameCommand - Sent hostname lwa23 command /usr/local/bin/lwa23 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:53 [DEBUG   ] sendHostnameCommand - Sent hostname lwa18 command /usr/local/bin/lwa18 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:54 [DEBUG   ] sendHostnameCommand - Sent hostname lwa10 command /usr/local/bin/lwa10 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:54 [DEBUG   ] sendHostnameCommand - Sent hostname lwa32 command /usr/local/bin/lwa32 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:54 [DEBUG   ] sendHostnameCommand - Sent hostname lwa11 command /usr/local/bin/lwa11 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:54 [DEBUG   ] sendHostnameCommand - Sent hostname lwa30 command /usr/local/bin/lwa30 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:55 [DEBUG   ] sendHostnameCommand - Sent hostname lwa12 command /usr/local/bin/lwa12 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:55 [DEBUG   ] sendHostnameCommand - Sent hostname lwa8 command /usr/local/bin/lwa8 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:55 [DEBUG   ] sendHostnameCommand - Sent hostname lwa13 command /usr/local/bin/lwa13 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:55 [DEBUG   ] sendHostnameCommand - Sent hostname lwa26 command /usr/local/bin/lwa26 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:55 [DEBUG   ] sendHostnameCommand - Sent hostname lwa14 command /usr/local/bin/lwa14 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:56 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:56 [DEBUG   ] sendHostnameCommand - Sent hostname lwa22 command /usr/local/bin/lwa22 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:56 [DEBUG   ] sendHostnameCommand - Sent hostname lwa20 command /usr/local/bin/lwa20 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:56 [DEBUG   ] sendHostnameCommand - Sent hostname lwa1 command /usr/local/bin/lwa1 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:57 [DEBUG   ] sendHostnameCommand - Sent hostname lwa5 command /usr/local/bin/lwa5 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:57 [DEBUG   ] sendHostnameCommand - Sent hostname lwa25 command /usr/local/bin/lwa25 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:57 [DEBUG   ] sendHostnameCommand - Sent hostname lwa35 command /usr/local/bin/lwa35 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:57 [DEBUG   ] sendHostnameCommand - Sent hostname lwa19 command /usr/local/bin/lwa19 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:58 [DEBUG   ] sendHostnameCommand - Sent hostname lwa21 command /usr/local/bin/lwa21 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:58 [DEBUG   ] sendHostnameCommand - Sent hostname lwa6 command /usr/local/bin/lwa6 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:58 [DEBUG   ] sendHostnameCommand - Sent hostname lwa7 command /usr/local/bin/lwa7 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:58 [DEBUG   ] sendHostnameCommand - Sent hostname lwa33 command /usr/local/bin/lwa33 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:59 [DEBUG   ] sendHostnameCommand - Sent hostname lwa15 command /usr/local/bin/lwa15 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:59 [DEBUG   ] sendHostnameCommand - Sent hostname lwa28 command /usr/local/bin/lwa28 /lwa/software/dp_shutdown.sh
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa27', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16663
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa24', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16664
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa23', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16665
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa18', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16666
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa10', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16667
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa32', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16668
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa11', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16669
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa30', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16670
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa12', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16671
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa8', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16672
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa13', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16673
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa26', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16674
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa14', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16675
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa22', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16676
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa20', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16677
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa1', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16678
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa5', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16679
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa25', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16680
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa35', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16681
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa19', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16682
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa21', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16683
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa6', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16684
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa7', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16685
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa33', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16686
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa15', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16687
2018-11-19 20:32:59 [DEBUG   ] 2018-11-19 20:32:59 Starting ['/usr/local/bin/lwa28', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] pid 16688
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa27', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16663 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa24', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16664 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa23', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16665 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa18', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16666 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa10', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16667 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa32', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16668 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa11', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16669 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa30', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16670 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa12', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16671 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa8', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16672 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa13', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16673 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa26', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16674 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa14', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16675 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa22', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16676 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa20', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16677 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa1', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16678 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa5', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16679 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa25', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16680 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa35', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16681 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa19', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16682 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa21', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16683 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa6', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16684 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa7', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16685 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa33', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16686 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa15', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16687 exited with status 0
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 ['/usr/local/bin/lwa28', '/lwa/software/Start.sh', '/lwa/hardware/lwa_dp_top_2_1_1_49.bin', '/lwa/software'] Process 16688 exited with status 0
2018-11-19 20:33:11 [INFO    ] Finish loading DP1 firmware'/lwa/hardware/lwa_dp_top_2_1_1_49.bin'
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa27', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa24', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa23', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa18', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa10', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa32', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa11', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa30', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa12', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa8', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa13', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa26', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa14', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa22', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa20', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa1', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa5', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa25', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa35', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa19', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa21', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa6', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa7', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa33', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa15', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] spawn_process: at 2018-11-19 20:33:11 running ['/usr/local/bin/lwa28', '/lwa/software/dp1_init.sh']
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 Starting ['/usr/local/bin/lwa16', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] pid 16720
2018-11-19 20:33:11 [DEBUG   ] 2018-11-19 20:33:11 Starting ['/usr/local/bin/lwa34', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] pid 16721
2018-11-19 20:33:17 [DEBUG   ] 2018-11-19 20:33:17 ['/usr/local/bin/lwa16', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] Process 16720 exited with status 0
2018-11-19 20:33:17 [DEBUG   ] 2018-11-19 20:33:17 ['/usr/local/bin/lwa34', '/lwa/software/Start2.sh', '/lwa/hardware/lwa_drx_top_3_2_0_6.bin', '/lwa/software'] Process 16721 exited with status 0
2018-11-19 20:33:17 [INFO    ] Finish loading DP2 firmware '/lwa/hardware/lwa_drx_top_3_2_0_6.bin'
2018-11-19 20:33:17 [DEBUG   ] spawn_process: at 2018-11-19 20:33:17 running ['/usr/local/bin/lwa16', '/lwa/software/dp2_init.sh']
2018-11-19 20:33:17 [DEBUG   ] spawn_process: at 2018-11-19 20:33:17 running ['/usr/local/bin/lwa34', '/lwa/software/dp2_init.sh']
2018-11-19 20:33:17 [INFO    ] Setting Beam 1 xbars
2018-11-19 20:33:17 [DEBUG   ] board 11 xbars (bd11 to bd5 through bot5) - beam 1
2018-11-19 20:33:17 [DEBUG   ] board 5 xbars (bd5 from bd11 through bot1) - beam 1
2018-11-19 20:33:17 [DEBUG   ] board 5 xbars (bd5 to bd13 through bot5) - beam 1
2018-11-19 20:33:17 [DEBUG   ] board 13 xbars (bd13 from bd5 through bot1) - beam 1
2018-11-19 20:33:17 [DEBUG   ] board 13 xbars (bd13 to bd7 through bot5) - beam 1
2018-11-19 20:33:18 [DEBUG   ] board 7 xbars (bd7 from bd13 through bot1) - beam 1
2018-11-19 20:33:18 [DEBUG   ] board 7 xbars (bd7 to bd9 through bot5) - beam 1
2018-11-19 20:33:18 [DEBUG   ] board 9 xbars (bd9 from bd7 through bot1) - beam 1
2018-11-19 20:33:18 [DEBUG   ] board 9 xbars (bd9 to bd3 through bot5) - beam 1
2018-11-19 20:33:18 [DEBUG   ] board 3 xbars (bd3 from bd9 through bot1) - beam 1
2018-11-19 20:33:18 [DEBUG   ] board 3 xbars (bd3 to bd2 through bot5) - beam 1
2018-11-19 20:33:18 [DEBUG   ] board 2 xbars (bd2 from bd3 through bot1) - beam 1
2018-11-19 20:33:18 [DEBUG   ] board 2 xbars (bd2 to bd6 through bot5) - beam 1
2018-11-19 20:33:18 [DEBUG   ] board 6 xbars (bd6 from bd2 through bot1) - beam 1
2018-11-19 20:33:19 [DEBUG   ] board 6 xbars (bd6 to bd12 through bot5) - beam 1
2018-11-19 20:33:19 [DEBUG   ] board 12 xbars (bd12 from bd6 through bot1) - beam 1
2018-11-19 20:33:19 [DEBUG   ] board 12 xbars (bd12 to bd8 through bot5) - beam 1
2018-11-19 20:33:19 [DEBUG   ] board 8 xbars (bd8 from bd12 through bot1) - beam 1
2018-11-19 20:33:19 [DEBUG   ] board 8 xbars (bd8 to bd4 through bot5) - beam 1
2018-11-19 20:33:19 [DEBUG   ] board 4 xbars (bd4 from bd8 through bot1) - beam 1
2018-11-19 20:33:19 [DEBUG   ] board 4 xbars (bd4 to bd10 through bot5) - beam 1
2018-11-19 20:33:19 [DEBUG   ] board 10 xbars (bd10 from bd4 through bot1) - beam 1
2018-11-19 20:33:19 [DEBUG   ] board 10 xbars (bd10 to bd14 through bot5) - beam 1
2018-11-19 20:33:20 [DEBUG   ] board 14 xbars (bd14 from bd10 through top1) - beam 1
2018-11-19 20:33:20 [DEBUG   ] board 28 xbars  (bd 28 to bd24 through top1)- beam 1
2018-11-19 20:33:20 [DEBUG   ] board 24 xbars (bd24 from bd28 through bot5) - beam 1
2018-11-19 20:33:20 [DEBUG   ] board 24 xbars (bd24 to bd18 through bot1) - beam 1
2018-11-19 20:33:20 [DEBUG   ] board 18 xbars (bd18 from bd24 through bot5) - beam 1
2018-11-19 20:33:20 [DEBUG   ] board 18 xbars (bd18 to bd22 through bot1) - beam 1
2018-11-19 20:33:20 [DEBUG   ] board 22 xbars (bd22 from bd18 through bot5) - beam 1
2018-11-19 20:33:20 [DEBUG   ] board 22 xbars (bd22 to bd26 through bot1) - beam 1
2018-11-19 20:33:20 [DEBUG   ] board 26 xbars (bd26 from bd22 through bot5) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 26 xbars (bd26 to bd20 through bot1) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 20 xbars (bd20 from bd26 through bot5) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 20 xbars (bd20 to bd16 through bot1) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 16 xbars (bd16 from bd20 through bot5) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 16 xbars (bd16 to bd17 through bot1) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 17 xbars (bd17 from bd16 through bot5) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 17 xbars (bd17 to bd23 through bot1) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 23 xbars (bd23 from bd17 through bot5) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 23 xbars (bd23 to bd21 through bot1) - beam 1
2018-11-19 20:33:21 [DEBUG   ] board 21 xbars (bd21 from bd23 through bot5) - beam 1
2018-11-19 20:33:22 [DEBUG   ] board 21 xbars (bd21 to bd27 through bot1) - beam 1
2018-11-19 20:33:22 [DEBUG   ] board 27 xbars (bd27 from bd21 through bot5) - beam 1
2018-11-19 20:33:22 [DEBUG   ] board 27 xbars (bd27 to bd19 through bot1) - beam 1
2018-11-19 20:33:22 [DEBUG   ] board 19 xbars (bd19 from bd27 through bot5) - beam 1
2018-11-19 20:33:22 [DEBUG   ] board 19 xbars (bd19 to bd25 through bot1) - beam 1
2018-11-19 20:33:22 [DEBUG   ] board 25 xbars (bd25 from bd19 through bot5) - beam 1
2018-11-19 20:33:22 [DEBUG   ] board 25 xbars (bd25 to bd15 through bot1) - beam 1
2018-11-19 20:33:22 [DEBUG   ] board 15 xbars (board 15 from board 25 through bot5) - beam 1
2018-11-19 20:33:22 [INFO    ] Setting Beam 2 xbars
2018-11-19 20:33:22 [DEBUG   ] board 3 xbars (bd3 to bd7 through top1) - beam 2
2018-11-19 20:33:23 [DEBUG   ] board 7 xbars (bd7 from bd3 through top5) - beam 2
2018-11-19 20:33:23 [DEBUG   ] board 7 xbars (bd7 to bd11 through top1) - beam 2
2018-11-19 20:33:23 [DEBUG   ] board 11 xbars (bd11 from bd7 through top5) - beam 2
2018-11-19 20:33:23 [DEBUG   ] board 11 xbars (bd11 to bd13 through top1) - beam 2
2018-11-19 20:33:23 [DEBUG   ] board 13 xbars (bd13 from bd11 through top5) - beam 2
2018-11-19 20:33:23 [DEBUG   ] board 13 xbars (bd13 to bd9 through top1) - beam 2
2018-11-19 20:33:23 [DEBUG   ] board 9 xbars (bd9 from bd13 through top5) - beam 2
2018-11-19 20:33:23 [DEBUG   ] board 9 xbars (bd9 to bd5 through top1) - beam 2
2018-11-19 20:33:23 [DEBUG   ] board 5 xbars (bd5 from bd9 through top5) - beam 2
2018-11-19 20:33:24 [DEBUG   ] board 5 xbars (bd5 to bd2 through top1) - beam 2
2018-11-19 20:33:24 [DEBUG   ] board 2 xbars (bd2 from bd5 through top5) - beam 2
2018-11-19 20:33:24 [DEBUG   ] board 2 xbars (bd2 to bd4 through top1) - beam 2
2018-11-19 20:33:24 [DEBUG   ] board 4 xbars (bd4 from bd2 through top5) - beam 2
2018-11-19 20:33:24 [DEBUG   ] board 4 xbars (bd4 to bd6 through top1) - beam 2
2018-11-19 20:33:24 [DEBUG   ] board 6 xbars (bd6 from bd4 through top5) - beam 2
2018-11-19 20:33:24 [DEBUG   ] board 6 xbars (bd6 to bd8 through top1) - beam 2
2018-11-19 20:33:24 [DEBUG   ] board 8 xbars (bd8 from bd6 through top5) - beam 2
2018-11-19 20:33:24 [DEBUG   ] board 8 xbars (bd8 to bd10 through top1) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 10 xbars (bd10 from bd8 through top5) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 10 xbars (bd10 to bd12 through top1) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 12 xbars (bd12 from bd10 through top5) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 12 xbars (bd12 to bd14 through top1) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 14 xbars (bd14 from bd12 through bot5) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 28 xbars (bd28 to bd26 through bot5) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 26 xbars (bd26 from bd28 through top1) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 26 xbars (bd26 to bd24 through top5) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 24 xbars (bd24 from bd26 through top1) - beam 2
2018-11-19 20:33:25 [DEBUG   ] board 24 xbars (bd24 to bd22 through top5) - beam 2
2018-11-19 20:33:26 [DEBUG   ] board 22 xbars (bd22 from bd24 through top1) - beam 2
2018-11-19 20:33:26 [DEBUG   ] board 22 xbars (bd22 to bd20 through top5) - beam 2
2018-11-19 20:33:26 [DEBUG   ] board 20 xbars (bd20 from bd22 through top1) - beam 2
2018-11-19 20:33:26 [DEBUG   ] board 20 xbars (bd20 to bd18 through top5) - beam 2
2018-11-19 20:33:26 [DEBUG   ] board 18 xbars (bd18 from bd20 through top1) - beam 2
2018-11-19 20:33:26 [DEBUG   ] board 18 xbars (bd18 to bd16 through top5) - beam 2
2018-11-19 20:33:26 [DEBUG   ] board 16 xbars (bd16 from bd18 through top1) - beam 2
2018-11-19 20:33:26 [DEBUG   ] board 16 xbars (bd16 to bd19 through top5) - beam 2
2018-11-19 20:33:26 [DEBUG   ] board 19 xbars (bd19 from bd16 through top1) - beam 2
2018-11-19 20:33:27 [DEBUG   ] board 19 xbars (bd19 to bd23 through top5) - beam 2
2018-11-19 20:33:27 [DEBUG   ] board 23 xbars (bd23 from bd19 through top1) - beam 2
2018-11-19 20:33:27 [DEBUG   ] board 23 xbars (bd23 to bd27 through top5) - beam 2
2018-11-19 20:33:27 [DEBUG   ] board 27 xbars (bd27 from bd23 through top1) - beam 2
2018-11-19 20:33:27 [DEBUG   ] board 27 xbars (bd27 to bd25 through top5) - beam 2
2018-11-19 20:33:27 [DEBUG   ] board 25 xbars (bd25 from bd27 through top1) - beam 2
2018-11-19 20:33:27 [DEBUG   ] board 25 xbars (bd25 to bd21 through top5) - beam 2
2018-11-19 20:33:27 [DEBUG   ] board 21 xbars (bd21 from bd25 through top1) - beam 2
2018-11-19 20:33:27 [DEBUG   ] board 21 xbars (bd21 to bd17 through top5) - beam 2
2018-11-19 20:33:28 [DEBUG   ] board 17 xbars (bd17 from bd21 through top1) - beam 2
2018-11-19 20:33:28 [DEBUG   ] board 17 xbars (bd17 to bd15 through top5) - beam 2
2018-11-19 20:33:28 [DEBUG   ] board 15 xbars (bd15 from bd17 through top1) - beam 2
2018-11-19 20:33:28 [INFO    ] Setting Beam 3 xbars
2018-11-19 20:33:28 [DEBUG   ] board 25 xbars (bd25 to bd19 through bot5) - beam 3
2018-11-19 20:33:28 [DEBUG   ] board 19 xbars (bd19 from bd25 through bot1) - beam 3
2018-11-19 20:33:28 [DEBUG   ] board 19 xbars (bd19 to bd27 through bot5) - beam 3
2018-11-19 20:33:28 [DEBUG   ] board 27 xbars (bd27 from bd19 through bot1) - beam 3
2018-11-19 20:33:28 [DEBUG   ] board 27 xbars (bd27 to bd21 through bot5) - beam 3
2018-11-19 20:33:28 [DEBUG   ] board 21 xbars (bd21 from bd27 through bot1) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 21 xbars (bd21 to bd23 through bot5) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 23 xbars (bd23 from bd21 through bot1) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 23 xbars (bd23 to bd17 through bot5) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 17 xbars (bd17 from bd23 through bot1) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 17 xbars (bd17 to bd16 through bot5) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 16 xbars (bd16 from bd17 through bot1) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 16 xbars (bd16 to bd20 through bot5) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 20 xbars (bd20 from bd16 through bot1) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 20 xbars (bd20 to bd26 through bot5) - beam 3
2018-11-19 20:33:29 [DEBUG   ] board 26 xbars (bd26 from bd20 through bot1) - beam 3
2018-11-19 20:33:30 [DEBUG   ] board 26 xbars (bd26 to bd22 through bot5) - beam 3
2018-11-19 20:33:30 [DEBUG   ] board 22 xbars (bd22 from bd26 through bot1) - beam 3
2018-11-19 20:33:30 [DEBUG   ] board 22 xbars (bd22 to bd18 through bot5) - beam 3
2018-11-19 20:33:30 [DEBUG   ] board 18 xbars (bd18 from bd22 through bot1) - beam 3
2018-11-19 20:33:30 [DEBUG   ] board 18 xbars (bd18 to bd24 through bot5) - beam 3
2018-11-19 20:33:30 [DEBUG   ] board 24 xbars (bd24 from bd18 through bot1) - beam 3
2018-11-19 20:33:30 [DEBUG   ] board 24 xbars (bd24 to bd28 through bot5) - beam 3
2018-11-19 20:33:30 [DEBUG   ] board 28 xbars (bd28 from bd24 through top1) - beam 3
2018-11-19 20:33:30 [DEBUG   ] board 14 xbars (bd14 to bd10 through top1) - beam 3
2018-11-19 20:33:31 [DEBUG   ] board 10 xbars (bd10 from bd14 through bot5) - beam 3
2018-11-19 20:33:31 [DEBUG   ] board 10 xbars (bd10 to bd4 through bot1) - beam 3
2018-11-19 20:33:31 [DEBUG   ] board 4 xbars (bd4 from bd10 through bot5) - beam 3
2018-11-19 20:33:31 [DEBUG   ] board 4 xbars (bd4 to bd8 through bot1) - beam 3
2018-11-19 20:33:31 [DEBUG   ] board 8 xbars (bd8 from bd4 through bot5) - beam 3
2018-11-19 20:33:31 [DEBUG   ] board 8 xbars (bd8 to bd12 through bot1) - beam 3
2018-11-19 20:33:31 [DEBUG   ] board 12 xbars (bd12 from bd8 through bot5) - beam 3
2018-11-19 20:33:31 [DEBUG   ] board 12 xbars (bd12 to bd6 through bot1) - beam 3
2018-11-19 20:33:31 [DEBUG   ] board 6 xbars (bd6 from bd12 through bot5) - beam 3
2018-11-19 20:33:32 [DEBUG   ] board 6 xbars (bd6 to bd2 through bot1) - beam 3
2018-11-19 20:33:32 [DEBUG   ] board 2 xbars (bd2 from bd6 through bot5) - beam 3
2018-11-19 20:33:32 [DEBUG   ] board 2 xbars (bd2 to bd3 through bot1) - beam 3
2018-11-19 20:33:32 [DEBUG   ] board 3 xbars (bd3 from bd2 through bot5) - beam 3
2018-11-19 20:33:32 [DEBUG   ] board 3 xbars (bd3 to bd9 through bot1) - beam 3
2018-11-19 20:33:32 [DEBUG   ] board 9 xbars (bd9 from bd3 through bot5) - beam 3
2018-11-19 20:33:32 [DEBUG   ] board 9 xbars (bd9 to bd7 through bot1) - beam 3
2018-11-19 20:33:32 [DEBUG   ] board 7 xbars (bd7 from bd9 through bot5) - beam 3
2018-11-19 20:33:32 [DEBUG   ] board 7 xbars (bd7 to bd13 through bot1) - beam 3
2018-11-19 20:33:33 [DEBUG   ] board 13 xbars (bd13 from bd7 through bot5) - beam 3
2018-11-19 20:33:33 [DEBUG   ] board 13 xbars (bd13 to bd5 through bot1) - beam 3
2018-11-19 20:33:33 [DEBUG   ] board 5 xbars (bd5 from bd13 through bot5) - beam 3
2018-11-19 20:33:33 [DEBUG   ] board 5 xbars (bd5 to bd11 through bot1) - beam 3
2018-11-19 20:33:33 [DEBUG   ] board 11 xbars (bd11 from bd5 through bot5) - beam 3
2018-11-19 20:33:33 [DEBUG   ] board 11 xbars (bd11 to bd1 through bot1) - beam 3
2018-11-19 20:33:33 [DEBUG   ] board 1 xbars (bd1 from bd11 through bot5) - beam 3
2018-11-19 20:33:33 [INFO    ] Setting Beam 4 xbars
2018-11-19 20:33:33 [DEBUG   ] board 17 xbars (bd17 to bd21 through top1) - beam 4
2018-11-19 20:33:33 [DEBUG   ] board 21 xbars (bd21 from bd17 through top5) - beam 4
2018-11-19 20:33:33 [DEBUG   ] board 21 xbars (bd21 to bd25 through top1) - beam 4
2018-11-19 20:33:34 [DEBUG   ] board 21 xbars (bd21 from bd17 through top5) - beam 4
2018-11-19 20:33:34 [DEBUG   ] board 21 xbars (bd21 to bd25 through top1) - beam 4
2018-11-19 20:33:34 [DEBUG   ] board 25 xbars (bd25 from bd21 through top5) - beam 4
2018-11-19 20:33:34 [DEBUG   ] board 25 xbars (bd25 to bd27 through top1) - beam 4
2018-11-19 20:33:34 [DEBUG   ] board 27 xbars (bd27 from bd25 through top5) - beam 4
2018-11-19 20:33:34 [DEBUG   ] board 27 xbars (bd27 to bd23 through top1) - beam 4
2018-11-19 20:33:34 [DEBUG   ] board 23 xbars (bd23 from bd27 through top5) - beam 4
2018-11-19 20:33:34 [DEBUG   ] board 23 xbars (bd23 to bd19 through top1) - beam 4
2018-11-19 20:33:34 [DEBUG   ] board 19 xbars (bd19 from bd23 through top5) - beam 4
2018-11-19 20:33:35 [DEBUG   ] board 19 xbars (bd19 to bd16 through top1) - beam 4
2018-11-19 20:33:35 [DEBUG   ] board 16 xbars (bd16 from bd19 through top5) - beam 4
2018-11-19 20:33:35 [DEBUG   ] board 16 xbars (bd16 to bd18 through top1) - beam 4
2018-11-19 20:33:35 [DEBUG   ] board 18 xbars (bd18 from bd16 through top5) - beam 4
2018-11-19 20:33:35 [DEBUG   ] board 18 xbars (bd18 to bd20 through top1) - beam 4
2018-11-19 20:33:35 [DEBUG   ] board 20 xbars (bd20 from bd18 through top5) - beam 4
2018-11-19 20:33:35 [DEBUG   ] board 20 xbars (bd20 to bd22 through top1) - beam 4
2018-11-19 20:33:35 [DEBUG   ] board 22 xbars (bd22 from bd20 through top5) - beam 4
2018-11-19 20:33:35 [DEBUG   ] board 22 xbars (bd22 to bd24 through top1) - beam 4
2018-11-19 20:33:36 [DEBUG   ] board 24 xbars (bd24 from bd22 through top5) - beam 4
2018-11-19 20:33:36 [DEBUG   ] board 24 xbars (bd24 to bd26 through top1) - beam 4
2018-11-19 20:33:36 [DEBUG   ] board 26 xbars (bd26 from bd24 through top5) - beam 4
2018-11-19 20:33:36 [DEBUG   ] board 26 xbars (bd26 to bd28 through top1) - beam 4
2018-11-19 20:33:36 [DEBUG   ] board 28 xbars (bd28 from bd26 through bot5) - beam 4
2018-11-19 20:33:36 [DEBUG   ] board 14 xbars (bd14 to bd12 through bot5) - beam 4
2018-11-19 20:33:36 [DEBUG   ] board 12 xbars (bd12 from bd14 through top1) - beam 4
2018-11-19 20:33:36 [DEBUG   ] board 12 xbars (bd12 to bd10 through top5) - beam 4
2018-11-19 20:33:36 [DEBUG   ] board 10 xbars (bd10 from bd12 through top1) - beam 4
2018-11-19 20:33:37 [DEBUG   ] board 10 xbars (bd10 to bd8 through top5) - beam 4
2018-11-19 20:33:37 [DEBUG   ] board 8 xbars (bd8 from bd10 through top1) - beam 4
2018-11-19 20:33:37 [DEBUG   ] board 8 xbars (bd8 to bd6 through top5) - beam 4
2018-11-19 20:33:37 [DEBUG   ] board 6 xbars (bd6 from bd8 through top1) - beam 4
2018-11-19 20:33:37 [DEBUG   ] board 6 xbars (bd6 to bd4 through top5) - beam 4
2018-11-19 20:33:37 [DEBUG   ] board 4 xbars (bd4 from bd6 through top1) - beam 4
2018-11-19 20:33:37 [DEBUG   ] board 4 xbars (bd4 to bd2 through top5) - beam 4
2018-11-19 20:33:37 [DEBUG   ] board 2 xbars (bd2 from bd4 through top1) - beam 4
2018-11-19 20:33:37 [DEBUG   ] board 2 xbars (bd2 to bd5 through top5) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 5 xbars (bd5 from bd2 through top1) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 5 xbars (bd5 to bd9 through top5) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 9 xbars (bd9 from bd5 through top1) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 9 xbars (bd9 to bd13 through top5) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 13 xbars (bd13 from bd9 through top1) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 13 xbars (bd13 to bd11 through top5) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 11 xbars (bd11 from bd13 through top1) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 11 xbars (bd11 to bd7 through top5) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 7 xbars (bd7 from bd11 through top1) - beam 4
2018-11-19 20:33:38 [DEBUG   ] board 7 xbars (bd7 to bd3 through top5) - beam 4
2018-11-19 20:33:39 [DEBUG   ] board 3 xbars (bd3 from bd7 through top1) - beam 4
2018-11-19 20:33:39 [DEBUG   ] board 3 xbars (bd3 to bd1 through top5) - beam 4
2018-11-19 20:33:39 [DEBUG   ] board 1 xbars (bd1 from bd3 through top1) - beam 4
2018-11-19 20:33:39 [INFO    ] All xbars set
2018-11-19 20:33:39 [INFO    ] Setting adder chain routing for all beams
2018-11-19 20:33:44 [INFO    ] Setting beam ids
2018-11-19 20:33:54 [INFO    ] Starting beam delay calibration
2018-11-19 20:34:06 [INFO    ] Beam 1 latency is 5744
2018-11-19 20:34:06 [INFO    ] Beam 2 latency is 5733
2018-11-19 20:34:06 [INFO    ] Beam 3 latency is 5738
2018-11-19 20:34:06 [INFO    ] Beam 4 latency is 5742
2018-11-19 20:34:06 [INFO    ] Stop calibration
2018-11-19 20:34:14 [INFO    ] Beam delay calibration successful
2018-11-19 20:34:14 [INFO    ] Calibration passed
2018-11-19 20:34:14 [DEBUG   ] spawn_process: at 2018-11-19 20:34:14 running ['/usr/local/bin/lwa34', '/lwa/software/cfg_eth 4 1 192.168.0.1 192.168.0.33 10000']
2018-11-19 20:34:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 5 0x138 8909
2018-11-19 20:34:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 5 0x13c 9111
2018-11-19 20:34:14 [DEBUG   ] spawn_process: at 2018-11-19 20:34:14 running ['/usr/local/bin/lwa34', '/lwa/software/cfg_eth 2 1 192.168.0.2 192.168.0.34 10000']
2018-11-19 20:34:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 1 0x138 8a0a
2018-11-19 20:34:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 /ape/core/xwl 1 0x13c 9212
2018-11-19 20:34:14 [DEBUG   ] spawn_process: at 2018-11-19 20:34:14 running ['/usr/local/bin/lwa16', '/lwa/software/cfg_eth 4 1 192.168.0.3 192.168.0.35 10000']
2018-11-19 20:34:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 5 0x138 8b0b
2018-11-19 20:34:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 5 0x13c 9313
2018-11-19 20:34:14 [DEBUG   ] spawn_process: at 2018-11-19 20:34:14 running ['/usr/local/bin/lwa16', '/lwa/software/cfg_eth 2 1 192.168.0.4 192.168.0.36 10000']
2018-11-19 20:34:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 1 0x138 8c0c
2018-11-19 20:34:14 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 /ape/core/xwl 1 0x13c 9414
2018-11-19 20:34:14 [DEBUG   ] SetDRXTime: DP2 - DP/MCS NTP skew is -3 msec
2018-11-19 20:34:14 [DEBUG   ] Got command RPT from MCS: ref #562102
2018-11-19 20:34:14 [DEBUG   ] info = Calibration passed
2018-11-19 20:34:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562102  26 58441 74054917 ABOOTINGCalibration passed'
2018-11-19 20:34:15 [DEBUG   ] SetDRXTime: started set_drx_time on hostname lwa16 at 286 ms (1542659655.289 s) with PID 18301 and time 1542659656
2018-11-19 20:34:15 [DEBUG   ] SetDRXTime: DP2 - DP/MCS NTP skew is -2 msec
2018-11-19 20:34:15 [DEBUG   ] SetDRXTime: started set_drx_time on hostname lwa34 at 375 ms (1542659655.377 s) with PID 18304 and time 1542659656
2018-11-19 20:34:16 [DEBUG   ] Finished updating ant stats in 0.870 seconds
2018-11-19 20:34:16 [INFO    ] Started antennas statistics background thread
2018-11-19 20:34:16 [INFO    ] Started board flags background thread
2018-11-19 20:34:16 [INFO    ] Started BAM background thread
2018-11-19 20:34:16 [INFO    ] Finished the INI process (ref. #562097) in 104.697 s
2018-11-19 20:34:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:34:17 [WARNING ] (13, 1, 'bot')
2018-11-19 20:34:17 [WARNING ] (13, 1, 'bot')
2018-11-19 20:34:17 [WARNING ] (13, 1, 'bot')
2018-11-19 20:34:17 [WARNING ] (13, 1, 'bot')
2018-11-19 20:34:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:34:17 [DEBUG   ] Finished updating board flags in 1.055 seconds
2018-11-19 20:34:28 [DEBUG   ] Got command TBN from MCS: ref #562103
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa27', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '1', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa24', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '2', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] tbn = exited with status True, code 0x00
2018-11-19 20:34:28 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_TBN   562103   8 58441 74068089 A  ERROR'
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa23', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '3', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa18', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '4', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa10', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '5', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa32', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '6', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa11', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '7', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa30', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '8', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa12', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '9', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa8', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '10', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa13', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '11', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa26', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '12', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa14', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '13', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa22', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '14', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa20', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '15', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa1', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '16', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa5', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '17', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa25', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '18', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa35', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '19', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa19', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '20', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa21', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '21', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa6', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '22', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa7', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '23', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa33', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '24', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa15', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '25', '1542659669']
2018-11-19 20:34:28 [DEBUG   ] spawn_process: at 2018-11-19 20:34:28 running ['/usr/local/bin/lwa28', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '26', '1542659669']
2018-11-19 20:34:28 [INFO    ] Finished spawning TBN in 0.048 s
2018-11-19 20:34:36 [DEBUG   ] Finished updating ant stats in 0.873 seconds
2018-11-19 20:34:56 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 20:34:59 [DEBUG   ] Got command RPT from MCS: ref #562104
2018-11-19 20:34:59 [DEBUG   ] summary = ERROR
2018-11-19 20:34:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562104  13 58441 74099769 A  ERRORERROR'
2018-11-19 20:34:59 [DEBUG   ] Got command RPT from MCS: ref #562105
2018-11-19 20:34:59 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:34:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562105  77 58441 74099973 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:35:00 [DEBUG   ] Got command RPT from MCS: ref #562106
2018-11-19 20:35:00 [DEBUG   ] summary = ERROR
2018-11-19 20:35:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562106  13 58441 74100165 A  ERRORERROR'
2018-11-19 20:35:00 [DEBUG   ] Got command RPT from MCS: ref #562107
2018-11-19 20:35:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:35:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562107  77 58441 74100369 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:35:00 [DEBUG   ] Got command RPT from MCS: ref #562108
2018-11-19 20:35:00 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:35:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562108  12 58441 74100573 A  ERRORB‰ÌÍ'
2018-11-19 20:35:16 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 20:35:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:35:17 [WARNING ] (13, 1, 'bot')
2018-11-19 20:35:17 [WARNING ] (13, 1, 'bot')
2018-11-19 20:35:17 [WARNING ] (13, 1, 'bot')
2018-11-19 20:35:17 [WARNING ] (13, 1, 'bot')
2018-11-19 20:35:17 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:35:17 [DEBUG   ] Finished updating board flags in 1.353 seconds
2018-11-19 20:35:22 [DEBUG   ] Got command STP from MCS: ref #1
2018-11-19 20:35:22 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 pkill tbn
2018-11-19 20:35:22 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP        1   8 58441 74122537 A  ERROR'
2018-11-19 20:35:22 [DEBUG   ] sendHostnameCommand - Sent hostname lwa27 command /usr/local/bin/lwa27 pkill tbn
2018-11-19 20:35:22 [DEBUG   ] sendHostnameCommand - Sent hostname lwa24 command /usr/local/bin/lwa24 pkill tbn
2018-11-19 20:35:22 [DEBUG   ] sendHostnameCommand - Sent hostname lwa23 command /usr/local/bin/lwa23 pkill tbn
2018-11-19 20:35:22 [DEBUG   ] sendHostnameCommand - Sent hostname lwa18 command /usr/local/bin/lwa18 pkill tbn
2018-11-19 20:35:22 [DEBUG   ] sendHostnameCommand - Sent hostname lwa10 command /usr/local/bin/lwa10 pkill tbn
2018-11-19 20:35:22 [DEBUG   ] sendHostnameCommand - Sent hostname lwa32 command /usr/local/bin/lwa32 pkill tbn
2018-11-19 20:35:22 [DEBUG   ] sendHostnameCommand - Sent hostname lwa11 command /usr/local/bin/lwa11 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa30 command /usr/local/bin/lwa30 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa12 command /usr/local/bin/lwa12 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa8 command /usr/local/bin/lwa8 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa13 command /usr/local/bin/lwa13 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa26 command /usr/local/bin/lwa26 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa14 command /usr/local/bin/lwa14 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa22 command /usr/local/bin/lwa22 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa20 command /usr/local/bin/lwa20 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa1 command /usr/local/bin/lwa1 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa5 command /usr/local/bin/lwa5 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa25 command /usr/local/bin/lwa25 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa35 command /usr/local/bin/lwa35 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa19 command /usr/local/bin/lwa19 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa21 command /usr/local/bin/lwa21 pkill tbn
2018-11-19 20:35:23 [DEBUG   ] sendHostnameCommand - Sent hostname lwa6 command /usr/local/bin/lwa6 pkill tbn
2018-11-19 20:35:24 [DEBUG   ] sendHostnameCommand - Sent hostname lwa7 command /usr/local/bin/lwa7 pkill tbn
2018-11-19 20:35:24 [DEBUG   ] sendHostnameCommand - Sent hostname lwa33 command /usr/local/bin/lwa33 pkill tbn
2018-11-19 20:35:24 [DEBUG   ] sendHostnameCommand - Sent hostname lwa15 command /usr/local/bin/lwa15 pkill tbn
2018-11-19 20:35:24 [DEBUG   ] sendHostnameCommand - Sent hostname lwa28 command /usr/local/bin/lwa28 pkill tbn
2018-11-19 20:35:24 [INFO    ] Stopped the currently running TBN process
2018-11-19 20:35:36 [DEBUG   ] Finished updating ant stats in 0.334 seconds
2018-11-19 20:35:46 [DEBUG   ] Got command RPT from MCS: ref #562109
2018-11-19 20:35:46 [DEBUG   ] summary = ERROR
2018-11-19 20:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562109  13 58441 74146501 A  ERRORERROR'
2018-11-19 20:35:46 [DEBUG   ] Got command RPT from MCS: ref #562110
2018-11-19 20:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562110  77 58441 74146502 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:35:57 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:36:17 [DEBUG   ] Finished updating ant stats in 1.233 seconds
2018-11-19 20:36:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:36:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:36:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:36:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:36:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:36:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:36:18 [DEBUG   ] Finished updating board flags in 1.848 seconds
2018-11-19 20:36:37 [DEBUG   ] Finished updating ant stats in 1.171 seconds
2018-11-19 20:36:57 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 20:37:17 [DEBUG   ] Finished updating ant stats in 1.031 seconds
2018-11-19 20:37:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:37:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:37:18 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 20:37:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:37:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:37:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:37:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:37:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:37:18 [DEBUG   ] Finished updating board flags in 1.805 seconds
2018-11-19 20:37:37 [DEBUG   ] Finished updating ant stats in 0.974 seconds
2018-11-19 20:37:57 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 20:38:17 [DEBUG   ] Finished updating ant stats in 0.833 seconds
2018-11-19 20:38:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:38:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:38:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:38:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:38:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:38:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:38:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:38:18 [DEBUG   ] Finished updating board flags in 1.785 seconds
2018-11-19 20:38:37 [DEBUG   ] Finished updating ant stats in 0.770 seconds
2018-11-19 20:38:57 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:39:17 [DEBUG   ] Finished updating ant stats in 0.631 seconds
2018-11-19 20:39:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:39:18 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:39:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:39:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:39:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:39:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:39:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:39:18 [DEBUG   ] Finished updating board flags in 1.617 seconds
2018-11-19 20:39:37 [DEBUG   ] Finished updating ant stats in 0.575 seconds
2018-11-19 20:39:57 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 20:39:59 [DEBUG   ] Got command RPT from MCS: ref #562132
2018-11-19 20:39:59 [DEBUG   ] summary = ERROR
2018-11-19 20:39:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562132  13 58441 74399853 A  ERRORERROR'
2018-11-19 20:40:00 [DEBUG   ] Got command RPT from MCS: ref #562133
2018-11-19 20:40:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:40:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562133  77 58441 74400057 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:40:00 [DEBUG   ] Got command RPT from MCS: ref #562134
2018-11-19 20:40:00 [DEBUG   ] summary = ERROR
2018-11-19 20:40:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562134  13 58441 74400249 A  ERRORERROR'
2018-11-19 20:40:00 [DEBUG   ] Got command RPT from MCS: ref #562136
2018-11-19 20:40:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:40:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562136  77 58441 74400453 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:40:00 [DEBUG   ] Got command RPT from MCS: ref #562138
2018-11-19 20:40:00 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:40:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562138  12 58441 74400657 A  ERRORB—™š'
2018-11-19 20:40:17 [DEBUG   ] Finished updating ant stats in 0.427 seconds
2018-11-19 20:40:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:40:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:40:18 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 20:40:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:40:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:40:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:40:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:40:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:40:18 [DEBUG   ] Finished updating board flags in 1.605 seconds
2018-11-19 20:40:37 [DEBUG   ] Finished updating ant stats in 0.358 seconds
2018-11-19 20:40:46 [DEBUG   ] Got command RPT from MCS: ref #562146
2018-11-19 20:40:46 [DEBUG   ] summary = ERROR
2018-11-19 20:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562146  13 58441 74446501 A  ERRORERROR'
2018-11-19 20:40:46 [DEBUG   ] Got command RPT from MCS: ref #562147
2018-11-19 20:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562147  77 58441 74446502 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:40:58 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:41:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:41:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:41:18 [DEBUG   ] Finished updating ant stats in 1.229 seconds
2018-11-19 20:41:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:41:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:41:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:41:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:41:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:41:18 [DEBUG   ] Finished updating board flags in 1.658 seconds
2018-11-19 20:41:38 [DEBUG   ] Finished updating ant stats in 1.175 seconds
2018-11-19 20:41:58 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 20:42:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:42:18 [DEBUG   ] Finished updating ant stats in 1.028 seconds
2018-11-19 20:42:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:42:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:42:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:42:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:42:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:42:18 [DEBUG   ] Finished updating board flags in 1.605 seconds
2018-11-19 20:42:38 [DEBUG   ] Finished updating ant stats in 0.974 seconds
2018-11-19 20:42:58 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 20:43:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:43:18 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 20:43:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:43:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:43:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:43:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:43:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:43:18 [DEBUG   ] Finished updating board flags in 1.759 seconds
2018-11-19 20:43:38 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 20:43:58 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:44:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:44:18 [DEBUG   ] Finished updating ant stats in 0.628 seconds
2018-11-19 20:44:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:44:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:44:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:44:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:44:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:44:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:44:18 [DEBUG   ] Finished updating board flags in 1.437 seconds
2018-11-19 20:44:38 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-19 20:44:58 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 20:44:59 [DEBUG   ] Got command RPT from MCS: ref #562167
2018-11-19 20:44:59 [DEBUG   ] summary = ERROR
2018-11-19 20:44:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562167  13 58441 74699945 A  ERRORERROR'
2018-11-19 20:45:00 [DEBUG   ] Got command RPT from MCS: ref #562168
2018-11-19 20:45:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:45:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562168  77 58441 74700137 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:45:00 [DEBUG   ] Got command RPT from MCS: ref #562169
2018-11-19 20:45:00 [DEBUG   ] summary = ERROR
2018-11-19 20:45:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562169  13 58441 74700341 A  ERRORERROR'
2018-11-19 20:45:00 [DEBUG   ] Got command RPT from MCS: ref #562170
2018-11-19 20:45:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:45:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562170  77 58441 74700545 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:45:00 [DEBUG   ] Got command RPT from MCS: ref #562171
2018-11-19 20:45:00 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:45:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562171  12 58441 74700749 A  ERRORBš™š'
2018-11-19 20:45:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:45:18 [DEBUG   ] Finished updating ant stats in 0.425 seconds
2018-11-19 20:45:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:45:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:45:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:45:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:45:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:45:18 [DEBUG   ] Finished updating board flags in 1.412 seconds
2018-11-19 20:45:38 [DEBUG   ] Finished updating ant stats in 0.374 seconds
2018-11-19 20:45:46 [DEBUG   ] Got command RPT from MCS: ref #562172
2018-11-19 20:45:46 [DEBUG   ] summary = ERROR
2018-11-19 20:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562172  13 58441 74746505 A  ERRORERROR'
2018-11-19 20:45:46 [DEBUG   ] Got command RPT from MCS: ref #562173
2018-11-19 20:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562173  77 58441 74746506 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:45:59 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:46:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:46:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:46:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:46:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:46:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:46:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:46:18 [DEBUG   ] Finished updating board flags in 1.370 seconds
2018-11-19 20:46:19 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:46:39 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:46:59 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:47:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:47:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:47:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:47:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:47:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:47:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:47:18 [DEBUG   ] Finished updating board flags in 1.230 seconds
2018-11-19 20:47:19 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 20:47:39 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:47:59 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:48:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:48:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 20:48:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:48:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:48:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:48:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:48:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:48:18 [DEBUG   ] Finished updating board flags in 1.183 seconds
2018-11-19 20:48:19 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:48:39 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:48:59 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:49:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:49:19 [DEBUG   ] Finished updating ant stats in 1.232 seconds
2018-11-19 20:49:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:49:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:49:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:49:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:49:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:49:19 [DEBUG   ] Finished updating board flags in 2.034 seconds
2018-11-19 20:49:39 [DEBUG   ] Finished updating ant stats in 1.174 seconds
2018-11-19 20:49:59 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 20:50:00 [DEBUG   ] Got command RPT from MCS: ref #562199
2018-11-19 20:50:00 [DEBUG   ] summary = ERROR
2018-11-19 20:50:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562199  13 58441 75000025 A  ERRORERROR'
2018-11-19 20:50:00 [DEBUG   ] Got command RPT from MCS: ref #562200
2018-11-19 20:50:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:50:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562200  77 58441 75000229 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:50:00 [DEBUG   ] Got command RPT from MCS: ref #562201
2018-11-19 20:50:00 [DEBUG   ] summary = ERROR
2018-11-19 20:50:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562201  13 58441 75000433 A  ERRORERROR'
2018-11-19 20:50:00 [DEBUG   ] Got command RPT from MCS: ref #562203
2018-11-19 20:50:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:50:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562203  77 58441 75000625 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:50:00 [DEBUG   ] Got command RPT from MCS: ref #562204
2018-11-19 20:50:00 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:50:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562204  12 58441 75000829 A  ERRORBš™š'
2018-11-19 20:50:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:50:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 20:50:19 [DEBUG   ] Finished updating ant stats in 1.031 seconds
2018-11-19 20:50:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:50:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:50:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:50:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:50:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:50:19 [DEBUG   ] Finished updating board flags in 2.013 seconds
2018-11-19 20:50:39 [DEBUG   ] Finished updating ant stats in 0.974 seconds
2018-11-19 20:50:46 [DEBUG   ] Got command RPT from MCS: ref #562209
2018-11-19 20:50:46 [DEBUG   ] summary = ERROR
2018-11-19 20:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562209  13 58441 75046501 A  ERRORERROR'
2018-11-19 20:50:46 [DEBUG   ] Got command RPT from MCS: ref #562210
2018-11-19 20:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562210  77 58441 75046502 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:50:59 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 20:51:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:51:19 [DEBUG   ] Finished updating ant stats in 0.797 seconds
2018-11-19 20:51:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:51:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:51:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:51:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:51:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:51:19 [DEBUG   ] Finished updating board flags in 2.163 seconds
2018-11-19 20:51:39 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 20:51:59 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:52:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:52:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:52:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:52:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:52:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:52:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:52:18 [DEBUG   ] Finished updating board flags in 1.250 seconds
2018-11-19 20:52:19 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:52:39 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:52:59 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:53:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:53:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:53:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:53:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:53:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:53:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:53:18 [DEBUG   ] Finished updating board flags in 1.210 seconds
2018-11-19 20:53:19 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:53:39 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:53:59 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:54:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:54:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:54:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:54:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:54:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:54:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:54:18 [DEBUG   ] Finished updating board flags in 1.181 seconds
2018-11-19 20:54:19 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:54:39 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:54:59 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:55:00 [DEBUG   ] Got command RPT from MCS: ref #562230
2018-11-19 20:55:00 [DEBUG   ] summary = ERROR
2018-11-19 20:55:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562230  13 58441 75300117 A  ERRORERROR'
2018-11-19 20:55:00 [DEBUG   ] Got command RPT from MCS: ref #562231
2018-11-19 20:55:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:55:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562231  77 58441 75300309 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:55:00 [DEBUG   ] Got command RPT from MCS: ref #562232
2018-11-19 20:55:00 [DEBUG   ] summary = ERROR
2018-11-19 20:55:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562232  13 58441 75300513 A  ERRORERROR'
2018-11-19 20:55:00 [DEBUG   ] Got command RPT from MCS: ref #562233
2018-11-19 20:55:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:55:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562233  77 58441 75300717 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:55:00 [DEBUG   ] Got command RPT from MCS: ref #562234
2018-11-19 20:55:00 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 20:55:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562234  12 58441 75300921 A  ERRORBš™š'
2018-11-19 20:55:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:55:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:55:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:55:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:55:18 [WARNING ] (13, 1, 'bot')
2018-11-19 20:55:18 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:55:18 [DEBUG   ] Finished updating board flags in 1.019 seconds
2018-11-19 20:55:19 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 20:55:39 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:55:46 [DEBUG   ] Got command RPT from MCS: ref #562239
2018-11-19 20:55:46 [DEBUG   ] summary = ERROR
2018-11-19 20:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562239  13 58441 75346501 A  ERRORERROR'
2018-11-19 20:55:46 [DEBUG   ] Got command RPT from MCS: ref #562240
2018-11-19 20:55:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 20:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562240  77 58441 75346502 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 20:55:59 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 20:56:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 20:56:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:56:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:56:19 [DEBUG   ] Finished updating ant stats in 0.630 seconds
2018-11-19 20:56:19 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 20:56:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:56:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:56:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:56:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:56:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:56:19 [DEBUG   ] Finished updating board flags in 1.594 seconds
2018-11-19 20:56:39 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-19 20:56:59 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 20:57:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:57:19 [DEBUG   ] Finished updating ant stats in 0.428 seconds
2018-11-19 20:57:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:57:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:57:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:57:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:57:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:57:19 [DEBUG   ] Finished updating board flags in 1.345 seconds
2018-11-19 20:57:39 [DEBUG   ] Finished updating ant stats in 0.378 seconds
2018-11-19 20:58:00 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:58:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:58:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:58:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:58:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:58:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:58:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:58:19 [DEBUG   ] Finished updating board flags in 1.206 seconds
2018-11-19 20:58:20 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:58:40 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:59:00 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:59:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 20:59:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 20:59:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:59:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:59:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:59:19 [WARNING ] (13, 1, 'bot')
2018-11-19 20:59:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 20:59:19 [DEBUG   ] Finished updating board flags in 1.180 seconds
2018-11-19 20:59:20 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 20:59:40 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:00:00 [DEBUG   ] Got command RPT from MCS: ref #562262
2018-11-19 21:00:00 [DEBUG   ] summary = ERROR
2018-11-19 21:00:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562262  13 58441 75600205 A  ERRORERROR'
2018-11-19 21:00:00 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:00:00 [DEBUG   ] Got command RPT from MCS: ref #562263
2018-11-19 21:00:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:00:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562263  77 58441 75600397 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:00:00 [DEBUG   ] Got command RPT from MCS: ref #562264
2018-11-19 21:00:00 [DEBUG   ] summary = ERROR
2018-11-19 21:00:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562264  13 58441 75600601 A  ERRORERROR'
2018-11-19 21:00:00 [DEBUG   ] Got command RPT from MCS: ref #562266
2018-11-19 21:00:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:00:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562266  77 58441 75600805 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:00:01 [DEBUG   ] Got command RPT from MCS: ref #562267
2018-11-19 21:00:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:00:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562267  12 58441 75601009 A  ERRORBš™š'
2018-11-19 21:00:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:00:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 21:00:20 [DEBUG   ] Finished updating ant stats in 1.234 seconds
2018-11-19 21:00:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:00:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:00:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:00:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:00:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:00:20 [DEBUG   ] Finished updating board flags in 2.088 seconds
2018-11-19 21:00:40 [DEBUG   ] Finished updating ant stats in 1.170 seconds
2018-11-19 21:00:46 [DEBUG   ] Got command RPT from MCS: ref #562272
2018-11-19 21:00:46 [DEBUG   ] summary = ERROR
2018-11-19 21:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562272  13 58441 75646509 A  ERRORERROR'
2018-11-19 21:00:46 [DEBUG   ] Got command RPT from MCS: ref #562273
2018-11-19 21:00:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562273  77 58441 75646510 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:01:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:01:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:01:20 [DEBUG   ] Finished updating ant stats in 1.033 seconds
2018-11-19 21:01:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 21:01:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:01:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:01:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:01:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:01:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:01:20 [DEBUG   ] Finished updating board flags in 1.968 seconds
2018-11-19 21:01:40 [DEBUG   ] Finished updating ant stats in 0.969 seconds
2018-11-19 21:02:00 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 21:02:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:02:20 [DEBUG   ] Finished updating ant stats in 0.825 seconds
2018-11-19 21:02:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:02:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:02:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:02:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:02:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:02:20 [DEBUG   ] Finished updating board flags in 1.799 seconds
2018-11-19 21:02:40 [DEBUG   ] Finished updating ant stats in 0.773 seconds
2018-11-19 21:03:00 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:03:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:03:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 21:03:19 [WARNING ] (13, 1, 'bot')
2018-11-19 21:03:19 [WARNING ] (13, 1, 'bot')
2018-11-19 21:03:19 [WARNING ] (13, 1, 'bot')
2018-11-19 21:03:19 [WARNING ] (13, 1, 'bot')
2018-11-19 21:03:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:03:19 [DEBUG   ] Finished updating board flags in 1.007 seconds
2018-11-19 21:03:20 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:03:40 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:04:00 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:04:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:04:19 [WARNING ] (13, 1, 'bot')
2018-11-19 21:04:19 [WARNING ] (13, 1, 'bot')
2018-11-19 21:04:19 [WARNING ] (13, 1, 'bot')
2018-11-19 21:04:19 [WARNING ] (13, 1, 'bot')
2018-11-19 21:04:19 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:04:19 [DEBUG   ] Finished updating board flags in 1.037 seconds
2018-11-19 21:04:20 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:04:40 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:05:00 [DEBUG   ] Finished updating ant stats in 0.562 seconds
2018-11-19 21:05:00 [DEBUG   ] Got command RPT from MCS: ref #562297
2018-11-19 21:05:00 [DEBUG   ] summary = ERROR
2018-11-19 21:05:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562297  13 58441 75900293 A  ERRORERROR'
2018-11-19 21:05:00 [DEBUG   ] Got command RPT from MCS: ref #562298
2018-11-19 21:05:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:05:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562298  77 58441 75900485 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:05:00 [DEBUG   ] Got command RPT from MCS: ref #562299
2018-11-19 21:05:00 [DEBUG   ] summary = ERROR
2018-11-19 21:05:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562299  13 58441 75900689 A  ERRORERROR'
2018-11-19 21:05:00 [DEBUG   ] Got command RPT from MCS: ref #562300
2018-11-19 21:05:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:05:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562300  77 58441 75900893 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:05:01 [DEBUG   ] Got command RPT from MCS: ref #562301
2018-11-19 21:05:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:05:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562301  12 58441 75901085 A  ERRORBš™š'
2018-11-19 21:05:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:05:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 21:05:20 [DEBUG   ] Finished updating ant stats in 0.430 seconds
2018-11-19 21:05:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 21:05:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:05:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:05:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:05:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:05:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:05:20 [DEBUG   ] Finished updating board flags in 1.415 seconds
2018-11-19 21:05:40 [DEBUG   ] Finished updating ant stats in 0.374 seconds
2018-11-19 21:05:46 [DEBUG   ] Got command RPT from MCS: ref #562302
2018-11-19 21:05:46 [DEBUG   ] summary = ERROR
2018-11-19 21:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562302  13 58441 75946505 A  ERRORERROR'
2018-11-19 21:05:46 [DEBUG   ] Got command RPT from MCS: ref #562303
2018-11-19 21:05:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562303  77 58441 75946506 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:06:01 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:06:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:06:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 21:06:21 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 21:06:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:06:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:06:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:06:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:06:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:06:21 [DEBUG   ] Finished updating board flags in 2.508 seconds
2018-11-19 21:06:41 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:07:01 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:07:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:07:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 21:07:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:07:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:07:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:07:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:07:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:07:20 [DEBUG   ] Finished updating board flags in 1.319 seconds
2018-11-19 21:07:21 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:07:41 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:08:01 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:08:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:08:21 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 21:08:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:08:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:08:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:08:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:08:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:08:21 [DEBUG   ] Finished updating board flags in 2.279 seconds
2018-11-19 21:08:41 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:09:01 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:09:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:09:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 21:09:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:09:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:09:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:09:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:09:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:09:20 [DEBUG   ] Finished updating board flags in 1.110 seconds
2018-11-19 21:09:21 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:09:41 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:10:00 [DEBUG   ] Got command RPT from MCS: ref #562325
2018-11-19 21:10:00 [DEBUG   ] summary = ERROR
2018-11-19 21:10:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562325  13 58441 76200369 A  ERRORERROR'
2018-11-19 21:10:00 [DEBUG   ] Got command RPT from MCS: ref #562326
2018-11-19 21:10:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:10:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562326  77 58441 76200573 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:10:00 [DEBUG   ] Got command RPT from MCS: ref #562327
2018-11-19 21:10:00 [DEBUG   ] summary = ERROR
2018-11-19 21:10:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562327  13 58441 76200777 A  ERRORERROR'
2018-11-19 21:10:00 [DEBUG   ] Got command RPT from MCS: ref #562329
2018-11-19 21:10:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:10:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562329  77 58441 76200969 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:10:01 [DEBUG   ] Got command RPT from MCS: ref #562331
2018-11-19 21:10:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:10:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562331  12 58441 76201173 A  ERRORB™™š'
2018-11-19 21:10:01 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:10:21 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 21:10:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:10:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:10:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:10:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:10:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:10:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:10:22 [DEBUG   ] Finished updating board flags in 2.977 seconds
2018-11-19 21:10:41 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 21:10:46 [DEBUG   ] Got command RPT from MCS: ref #562339
2018-11-19 21:10:46 [DEBUG   ] summary = ERROR
2018-11-19 21:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562339  13 58441 76246509 A  ERRORERROR'
2018-11-19 21:10:46 [DEBUG   ] Got command RPT from MCS: ref #562340
2018-11-19 21:10:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562340  77 58441 76246510 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:11:01 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 21:11:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:11:21 [DEBUG   ] Finished updating ant stats in 1.228 seconds
2018-11-19 21:11:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:11:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:11:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:11:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:11:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:11:21 [DEBUG   ] Finished updating board flags in 1.813 seconds
2018-11-19 21:11:41 [DEBUG   ] Finished updating ant stats in 1.166 seconds
2018-11-19 21:12:01 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 21:12:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:12:21 [DEBUG   ] Finished updating ant stats in 1.029 seconds
2018-11-19 21:12:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 21:12:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:12:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:12:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:12:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:12:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:12:21 [DEBUG   ] Finished updating board flags in 1.808 seconds
2018-11-19 21:12:41 [DEBUG   ] Finished updating ant stats in 0.973 seconds
2018-11-19 21:13:01 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 21:13:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:13:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:13:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:13:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:13:20 [WARNING ] (13, 1, 'bot')
2018-11-19 21:13:20 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:13:20 [DEBUG   ] Finished updating board flags in 1.214 seconds
2018-11-19 21:13:21 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 21:13:41 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 21:14:01 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 21:14:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:14:21 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 21:14:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:14:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:14:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:14:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:14:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:14:21 [DEBUG   ] Finished updating board flags in 1.720 seconds
2018-11-19 21:14:41 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:15:00 [DEBUG   ] Got command RPT from MCS: ref #562360
2018-11-19 21:15:00 [DEBUG   ] summary = ERROR
2018-11-19 21:15:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562360  13 58441 76500461 A  ERRORERROR'
2018-11-19 21:15:00 [DEBUG   ] Got command RPT from MCS: ref #562361
2018-11-19 21:15:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:15:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562361  77 58441 76500662 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:15:00 [DEBUG   ] Got command RPT from MCS: ref #562362
2018-11-19 21:15:00 [DEBUG   ] summary = ERROR
2018-11-19 21:15:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562362  13 58441 76500862 A  ERRORERROR'
2018-11-19 21:15:01 [DEBUG   ] Got command RPT from MCS: ref #562363
2018-11-19 21:15:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:15:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562363  77 58441 76501057 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:15:01 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 21:15:01 [DEBUG   ] Got command RPT from MCS: ref #562364
2018-11-19 21:15:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:15:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562364  12 58441 76501261 A  ERRORB™™š'
2018-11-19 21:15:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:15:21 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 21:15:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:15:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:15:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:15:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:15:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:15:21 [DEBUG   ] Finished updating board flags in 1.595 seconds
2018-11-19 21:15:41 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:15:46 [DEBUG   ] Got command RPT from MCS: ref #562365
2018-11-19 21:15:46 [DEBUG   ] summary = ERROR
2018-11-19 21:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562365  13 58441 76546505 A  ERRORERROR'
2018-11-19 21:15:46 [DEBUG   ] Got command RPT from MCS: ref #562366
2018-11-19 21:15:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562366  77 58441 76546506 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:16:01 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 21:16:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:16:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 21:16:21 [DEBUG   ] Finished updating ant stats in 0.629 seconds
2018-11-19 21:16:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:16:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:16:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:16:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:16:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:16:21 [DEBUG   ] Finished updating board flags in 1.473 seconds
2018-11-19 21:16:41 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-19 21:17:01 [DEBUG   ] Finished updating ant stats in 0.447 seconds
2018-11-19 21:17:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:17:21 [DEBUG   ] Finished updating ant stats in 0.385 seconds
2018-11-19 21:17:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:17:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:17:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:17:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:17:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:17:21 [DEBUG   ] Finished updating board flags in 1.390 seconds
2018-11-19 21:17:42 [DEBUG   ] Finished updating ant stats in 1.174 seconds
2018-11-19 21:18:02 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 21:18:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:18:21 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 21:18:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:18:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:18:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:18:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:18:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:18:21 [DEBUG   ] Finished updating board flags in 1.235 seconds
2018-11-19 21:18:22 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:18:42 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:19:02 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 21:19:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 21:19:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:19:22 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 21:19:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:19:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:19:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:19:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:19:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:19:22 [DEBUG   ] Finished updating board flags in 2.062 seconds
2018-11-19 21:19:42 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 21:20:00 [DEBUG   ] Got command RPT from MCS: ref #562392
2018-11-19 21:20:00 [DEBUG   ] summary = ERROR
2018-11-19 21:20:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562392  13 58441 76800541 A  ERRORERROR'
2018-11-19 21:20:00 [DEBUG   ] Got command RPT from MCS: ref #562393
2018-11-19 21:20:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:20:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562393  77 58441 76800745 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:20:00 [DEBUG   ] Got command RPT from MCS: ref #562394
2018-11-19 21:20:00 [DEBUG   ] summary = ERROR
2018-11-19 21:20:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562394  13 58441 76800949 A  ERRORERROR'
2018-11-19 21:20:01 [DEBUG   ] Got command RPT from MCS: ref #562396
2018-11-19 21:20:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:20:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562396  77 58441 76801141 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:20:01 [DEBUG   ] Got command RPT from MCS: ref #562397
2018-11-19 21:20:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:20:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562397  12 58441 76801345 A  ERRORB™™š'
2018-11-19 21:20:02 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 21:20:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:20:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:20:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:20:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:20:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:20:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:20:21 [DEBUG   ] Finished updating board flags in 1.091 seconds
2018-11-19 21:20:22 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 21:20:42 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 21:20:46 [DEBUG   ] Got command RPT from MCS: ref #562402
2018-11-19 21:20:46 [DEBUG   ] summary = ERROR
2018-11-19 21:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562402  13 58441 76846509 A  ERRORERROR'
2018-11-19 21:20:46 [DEBUG   ] Got command RPT from MCS: ref #562403
2018-11-19 21:20:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562403  77 58441 76846510 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:21:02 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-19 21:21:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:21:22 [DEBUG   ] Finished updating ant stats in 0.834 seconds
2018-11-19 21:21:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:21:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:21:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:21:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:21:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:21:22 [DEBUG   ] Finished updating board flags in 2.093 seconds
2018-11-19 21:21:42 [DEBUG   ] Finished updating ant stats in 0.769 seconds
2018-11-19 21:22:02 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:22:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:22:22 [DEBUG   ] Finished updating ant stats in 0.629 seconds
2018-11-19 21:22:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:22:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:22:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:22:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:22:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:22:22 [DEBUG   ] Finished updating board flags in 1.939 seconds
2018-11-19 21:22:42 [DEBUG   ] Finished updating ant stats in 0.573 seconds
2018-11-19 21:23:02 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-19 21:23:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:23:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 21:23:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:23:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:23:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:23:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:23:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:23:21 [DEBUG   ] Finished updating board flags in 1.246 seconds
2018-11-19 21:23:22 [DEBUG   ] Finished updating ant stats in 0.381 seconds
2018-11-19 21:23:43 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:24:03 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 21:24:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:24:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:24:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:24:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:24:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:24:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:24:21 [DEBUG   ] Finished updating board flags in 1.187 seconds
2018-11-19 21:24:23 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:24:43 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:25:00 [DEBUG   ] Got command RPT from MCS: ref #562423
2018-11-19 21:25:00 [DEBUG   ] summary = ERROR
2018-11-19 21:25:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562423  13 58441 77100629 A  ERRORERROR'
2018-11-19 21:25:00 [DEBUG   ] Got command RPT from MCS: ref #562424
2018-11-19 21:25:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:25:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562424  77 58441 77100833 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:25:01 [DEBUG   ] Got command RPT from MCS: ref #562425
2018-11-19 21:25:01 [DEBUG   ] summary = ERROR
2018-11-19 21:25:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562425  13 58441 77101025 A  ERRORERROR'
2018-11-19 21:25:01 [DEBUG   ] Got command RPT from MCS: ref #562426
2018-11-19 21:25:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:25:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562426  77 58441 77101229 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:25:01 [DEBUG   ] Got command RPT from MCS: ref #562427
2018-11-19 21:25:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:25:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562427  12 58441 77101433 A  ERRORB™™š'
2018-11-19 21:25:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:25:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:25:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 21:25:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:25:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:25:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:25:21 [WARNING ] (13, 1, 'bot')
2018-11-19 21:25:21 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:25:21 [DEBUG   ] Finished updating board flags in 0.937 seconds
2018-11-19 21:25:23 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:25:43 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:25:46 [DEBUG   ] Got command RPT from MCS: ref #562432
2018-11-19 21:25:46 [DEBUG   ] summary = ERROR
2018-11-19 21:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562432  13 58441 77146509 A  ERRORERROR'
2018-11-19 21:25:46 [DEBUG   ] Got command RPT from MCS: ref #562433
2018-11-19 21:25:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562433  77 58441 77146510 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:26:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:26:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 21:26:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 21:26:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:26:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 21:26:23 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 21:26:23 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 21:26:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:26:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:26:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:26:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:26:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:26:23 [DEBUG   ] Finished updating board flags in 2.276 seconds
2018-11-19 21:26:43 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:27:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:27:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:27:22 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 21:27:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:27:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:27:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:27:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:27:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:27:22 [DEBUG   ] Finished updating board flags in 1.392 seconds
2018-11-19 21:27:23 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:27:43 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:28:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:28:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:28:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:28:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:28:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:28:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:28:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:28:22 [DEBUG   ] Finished updating board flags in 1.140 seconds
2018-11-19 21:28:23 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-19 21:28:43 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:29:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:29:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:29:23 [DEBUG   ] Finished updating ant stats in 1.033 seconds
2018-11-19 21:29:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:29:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:29:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:29:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:29:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:29:23 [DEBUG   ] Finished updating board flags in 2.061 seconds
2018-11-19 21:29:43 [DEBUG   ] Finished updating ant stats in 0.970 seconds
2018-11-19 21:30:00 [DEBUG   ] Got command RPT from MCS: ref #562455
2018-11-19 21:30:00 [DEBUG   ] summary = ERROR
2018-11-19 21:30:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562455  13 58441 77400709 A  ERRORERROR'
2018-11-19 21:30:00 [DEBUG   ] Got command RPT from MCS: ref #562456
2018-11-19 21:30:00 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:30:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562456  77 58441 77400913 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:30:01 [DEBUG   ] Got command RPT from MCS: ref #562457
2018-11-19 21:30:01 [DEBUG   ] summary = ERROR
2018-11-19 21:30:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562457  13 58441 77401117 A  ERRORERROR'
2018-11-19 21:30:01 [DEBUG   ] Got command RPT from MCS: ref #562459
2018-11-19 21:30:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:30:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562459  77 58441 77401321 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:30:01 [DEBUG   ] Got command RPT from MCS: ref #562460
2018-11-19 21:30:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:30:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562460  12 58441 77401513 A  ERRORB™™š'
2018-11-19 21:30:03 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 21:30:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:30:23 [DEBUG   ] Finished updating ant stats in 0.825 seconds
2018-11-19 21:30:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:30:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:30:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:30:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:30:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:30:23 [DEBUG   ] Finished updating board flags in 1.962 seconds
2018-11-19 21:30:43 [DEBUG   ] Finished updating ant stats in 0.774 seconds
2018-11-19 21:30:46 [DEBUG   ] Got command RPT from MCS: ref #562465
2018-11-19 21:30:46 [DEBUG   ] summary = ERROR
2018-11-19 21:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562465  13 58441 77446513 A  ERRORERROR'
2018-11-19 21:30:46 [DEBUG   ] Got command RPT from MCS: ref #562466
2018-11-19 21:30:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562466  77 58441 77446514 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:31:03 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 21:31:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:31:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:31:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:31:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:31:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:31:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:31:22 [DEBUG   ] Finished updating board flags in 1.258 seconds
2018-11-19 21:31:23 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 21:31:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:32:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:32:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:32:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 21:32:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:32:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:32:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:32:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:32:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:32:22 [DEBUG   ] Finished updating board flags in 1.215 seconds
2018-11-19 21:32:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:32:43 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 21:33:03 [DEBUG   ] Finished updating ant stats in 0.597 seconds
2018-11-19 21:33:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:33:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:33:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:33:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:33:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:33:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:33:22 [DEBUG   ] Finished updating board flags in 1.130 seconds
2018-11-19 21:33:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 21:33:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 21:34:03 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-19 21:34:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:34:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:34:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:34:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:34:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:34:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:34:22 [DEBUG   ] Finished updating board flags in 1.034 seconds
2018-11-19 21:34:23 [DEBUG   ] Finished updating ant stats in 0.397 seconds
2018-11-19 21:34:44 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 21:35:00 [DEBUG   ] Got command RPT from MCS: ref #562490
2018-11-19 21:35:00 [DEBUG   ] summary = ERROR
2018-11-19 21:35:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562490  13 58441 77700797 A  ERRORERROR'
2018-11-19 21:35:01 [DEBUG   ] Got command RPT from MCS: ref #562491
2018-11-19 21:35:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:35:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562491  77 58441 77701001 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:35:01 [DEBUG   ] Got command RPT from MCS: ref #562492
2018-11-19 21:35:01 [DEBUG   ] summary = ERROR
2018-11-19 21:35:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562492  13 58441 77701205 A  ERRORERROR'
2018-11-19 21:35:01 [DEBUG   ] Got command RPT from MCS: ref #562493
2018-11-19 21:35:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:35:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562493  77 58441 77701397 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:35:01 [DEBUG   ] Got command RPT from MCS: ref #562494
2018-11-19 21:35:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:35:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562494  12 58441 77701601 A  ERRORB™™š'
2018-11-19 21:35:04 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:35:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:35:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 21:35:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:35:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:35:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:35:22 [WARNING ] (13, 1, 'bot')
2018-11-19 21:35:22 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:35:22 [DEBUG   ] Finished updating board flags in 0.954 seconds
2018-11-19 21:35:24 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:35:44 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:35:46 [DEBUG   ] Got command RPT from MCS: ref #562495
2018-11-19 21:35:46 [DEBUG   ] summary = ERROR
2018-11-19 21:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562495  13 58441 77746513 A  ERRORERROR'
2018-11-19 21:35:46 [DEBUG   ] Got command RPT from MCS: ref #562496
2018-11-19 21:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562496  77 58441 77746514 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:36:04 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:36:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:36:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:36:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:36:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:36:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:36:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:36:23 [DEBUG   ] Finished updating board flags in 1.405 seconds
2018-11-19 21:36:24 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:36:44 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:37:04 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:37:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 21:37:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:37:24 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 21:37:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:37:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:37:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:37:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:37:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:37:24 [DEBUG   ] Finished updating board flags in 2.498 seconds
2018-11-19 21:37:44 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:38:04 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 21:38:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:38:24 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 21:38:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:38:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:38:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:38:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:38:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:38:24 [DEBUG   ] Finished updating board flags in 2.439 seconds
2018-11-19 21:38:44 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:39:04 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-19 21:39:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 21:39:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:39:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:39:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:39:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:39:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:39:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:39:23 [DEBUG   ] Finished updating board flags in 1.280 seconds
2018-11-19 21:39:24 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:39:44 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:40:00 [DEBUG   ] Got command RPT from MCS: ref #562518
2018-11-19 21:40:00 [DEBUG   ] summary = ERROR
2018-11-19 21:40:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562518  13 58441 78000889 A  ERRORERROR'
2018-11-19 21:40:01 [DEBUG   ] Got command RPT from MCS: ref #562519
2018-11-19 21:40:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:40:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562519  77 58441 78001081 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:40:01 [DEBUG   ] Got command RPT from MCS: ref #562520
2018-11-19 21:40:01 [DEBUG   ] summary = ERROR
2018-11-19 21:40:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562520  13 58441 78001285 A  ERRORERROR'
2018-11-19 21:40:01 [DEBUG   ] Got command RPT from MCS: ref #562522
2018-11-19 21:40:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:40:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562522  77 58441 78001489 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:40:01 [DEBUG   ] Got command RPT from MCS: ref #562524
2018-11-19 21:40:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:40:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562524  12 58441 78001681 A  ERRORB™™š'
2018-11-19 21:40:04 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:40:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:40:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:40:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:40:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:40:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:40:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:40:23 [DEBUG   ] Finished updating board flags in 1.181 seconds
2018-11-19 21:40:24 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-19 21:40:44 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:40:46 [DEBUG   ] Got command RPT from MCS: ref #562532
2018-11-19 21:40:46 [DEBUG   ] summary = ERROR
2018-11-19 21:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562532  13 58441 78046513 A  ERRORERROR'
2018-11-19 21:40:46 [DEBUG   ] Got command RPT from MCS: ref #562533
2018-11-19 21:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562533  77 58441 78046514 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:41:04 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:41:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 21:41:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:41:24 [DEBUG   ] Finished updating ant stats in 1.029 seconds
2018-11-19 21:41:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:41:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:41:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:41:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:41:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:41:24 [DEBUG   ] Finished updating board flags in 2.032 seconds
2018-11-19 21:41:44 [DEBUG   ] Finished updating ant stats in 0.974 seconds
2018-11-19 21:42:04 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 21:42:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:42:24 [DEBUG   ] Finished updating ant stats in 0.829 seconds
2018-11-19 21:42:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:42:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:42:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:42:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:42:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:42:24 [DEBUG   ] Finished updating board flags in 2.022 seconds
2018-11-19 21:42:44 [DEBUG   ] Finished updating ant stats in 0.774 seconds
2018-11-19 21:43:04 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:43:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 21:43:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:43:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:43:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:43:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:43:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:43:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:43:23 [DEBUG   ] Finished updating board flags in 1.388 seconds
2018-11-19 21:43:24 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:43:44 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:44:04 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:44:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:44:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:44:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:44:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:44:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:44:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:44:23 [DEBUG   ] Finished updating board flags in 1.243 seconds
2018-11-19 21:44:24 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 21:44:44 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-19 21:45:00 [DEBUG   ] Got command RPT from MCS: ref #562553
2018-11-19 21:45:00 [DEBUG   ] summary = ERROR
2018-11-19 21:45:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562553  13 58441 78300973 A  ERRORERROR'
2018-11-19 21:45:01 [DEBUG   ] Got command RPT from MCS: ref #562554
2018-11-19 21:45:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:45:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562554  77 58441 78301169 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:45:01 [DEBUG   ] Got command RPT from MCS: ref #562555
2018-11-19 21:45:01 [DEBUG   ] summary = ERROR
2018-11-19 21:45:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562555  13 58441 78301373 A  ERRORERROR'
2018-11-19 21:45:01 [DEBUG   ] Got command RPT from MCS: ref #562556
2018-11-19 21:45:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:45:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562556  77 58441 78301577 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:45:01 [DEBUG   ] Got command RPT from MCS: ref #562557
2018-11-19 21:45:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:45:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562557  12 58441 78301769 A  ERRORB˜™š'
2018-11-19 21:45:04 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:45:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:45:24 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-19 21:45:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:45:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:45:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:45:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:45:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:45:24 [DEBUG   ] Finished updating board flags in 1.703 seconds
2018-11-19 21:45:44 [DEBUG   ] Finished updating ant stats in 0.398 seconds
2018-11-19 21:45:46 [DEBUG   ] Got command RPT from MCS: ref #562558
2018-11-19 21:45:46 [DEBUG   ] summary = ERROR
2018-11-19 21:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562558  13 58441 78346517 A  ERRORERROR'
2018-11-19 21:45:46 [DEBUG   ] Got command RPT from MCS: ref #562559
2018-11-19 21:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562559  77 58441 78346518 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:46:05 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:46:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 21:46:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 21:46:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:46:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 21:46:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:46:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:46:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:46:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:46:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:46:23 [DEBUG   ] Finished updating board flags in 1.068 seconds
2018-11-19 21:46:25 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:46:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:47:05 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:47:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:47:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:47:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:47:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:47:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:47:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:47:23 [DEBUG   ] Finished updating board flags in 1.046 seconds
2018-11-19 21:47:25 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:47:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:48:05 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:48:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 21:48:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:48:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:48:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:48:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:48:23 [WARNING ] (13, 1, 'bot')
2018-11-19 21:48:23 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:48:23 [DEBUG   ] Finished updating board flags in 0.949 seconds
2018-11-19 21:48:25 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:48:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:49:05 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:49:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 21:49:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:49:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 21:49:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:49:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:49:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:49:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:49:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:49:24 [DEBUG   ] Finished updating board flags in 1.399 seconds
2018-11-19 21:49:25 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:49:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:50:01 [DEBUG   ] Got command RPT from MCS: ref #562585
2018-11-19 21:50:01 [DEBUG   ] summary = ERROR
2018-11-19 21:50:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562585  13 58441 78601057 A  ERRORERROR'
2018-11-19 21:50:01 [DEBUG   ] Got command RPT from MCS: ref #562586
2018-11-19 21:50:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:50:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562586  77 58441 78601261 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:50:01 [DEBUG   ] Got command RPT from MCS: ref #562587
2018-11-19 21:50:01 [DEBUG   ] summary = ERROR
2018-11-19 21:50:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562587  13 58441 78601453 A  ERRORERROR'
2018-11-19 21:50:01 [DEBUG   ] Got command RPT from MCS: ref #562589
2018-11-19 21:50:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:50:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562589  77 58441 78601657 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:50:01 [DEBUG   ] Got command RPT from MCS: ref #562590
2018-11-19 21:50:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:50:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562590  12 58441 78601861 A  ERRORB˜™š'
2018-11-19 21:50:05 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:50:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:50:25 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 21:50:25 [WARNING ] (13, 1, 'bot')
2018-11-19 21:50:25 [WARNING ] (13, 1, 'bot')
2018-11-19 21:50:25 [WARNING ] (13, 1, 'bot')
2018-11-19 21:50:25 [WARNING ] (13, 1, 'bot')
2018-11-19 21:50:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:50:25 [DEBUG   ] Finished updating board flags in 2.305 seconds
2018-11-19 21:50:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:50:46 [DEBUG   ] Got command RPT from MCS: ref #562595
2018-11-19 21:50:46 [DEBUG   ] summary = ERROR
2018-11-19 21:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562595  13 58441 78646509 A  ERRORERROR'
2018-11-19 21:50:46 [DEBUG   ] Got command RPT from MCS: ref #562596
2018-11-19 21:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562596  77 58441 78646510 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:51:05 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 21:51:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:51:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 21:51:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:51:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:51:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:51:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:51:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:51:24 [DEBUG   ] Finished updating board flags in 1.071 seconds
2018-11-19 21:51:25 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 21:51:45 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 21:52:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 21:52:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:52:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 21:52:25 [DEBUG   ] Finished updating ant stats in 0.829 seconds
2018-11-19 21:52:25 [WARNING ] (13, 1, 'bot')
2018-11-19 21:52:25 [WARNING ] (13, 1, 'bot')
2018-11-19 21:52:25 [WARNING ] (13, 1, 'bot')
2018-11-19 21:52:25 [WARNING ] (13, 1, 'bot')
2018-11-19 21:52:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:52:25 [DEBUG   ] Finished updating board flags in 2.036 seconds
2018-11-19 21:52:45 [DEBUG   ] Finished updating ant stats in 0.774 seconds
2018-11-19 21:53:05 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 21:53:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 21:53:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:53:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:53:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:53:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:53:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:53:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:53:24 [DEBUG   ] Finished updating board flags in 1.419 seconds
2018-11-19 21:53:25 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 21:53:45 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 21:54:05 [DEBUG   ] Finished updating ant stats in 0.398 seconds
2018-11-19 21:54:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 21:54:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:54:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 21:54:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:54:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:54:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:54:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:54:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:54:24 [DEBUG   ] Finished updating board flags in 1.417 seconds
2018-11-19 21:54:26 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:54:46 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:55:01 [DEBUG   ] Got command RPT from MCS: ref #562616
2018-11-19 21:55:01 [DEBUG   ] summary = ERROR
2018-11-19 21:55:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562616  13 58441 78901137 A  ERRORERROR'
2018-11-19 21:55:01 [DEBUG   ] Got command RPT from MCS: ref #562617
2018-11-19 21:55:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:55:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562617  77 58441 78901341 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:55:01 [DEBUG   ] Got command RPT from MCS: ref #562618
2018-11-19 21:55:01 [DEBUG   ] summary = ERROR
2018-11-19 21:55:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562618  13 58441 78901545 A  ERRORERROR'
2018-11-19 21:55:01 [DEBUG   ] Got command RPT from MCS: ref #562619
2018-11-19 21:55:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:55:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562619  77 58441 78901741 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:55:01 [DEBUG   ] Got command RPT from MCS: ref #562620
2018-11-19 21:55:01 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 21:55:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562620  12 58441 78901945 A  ERRORB˜™š'
2018-11-19 21:55:06 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:55:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:55:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:55:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:55:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:55:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:55:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:55:24 [DEBUG   ] Finished updating board flags in 1.404 seconds
2018-11-19 21:55:26 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:55:46 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:55:46 [DEBUG   ] Got command RPT from MCS: ref #562625
2018-11-19 21:55:46 [DEBUG   ] summary = ERROR
2018-11-19 21:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562625  13 58441 78946517 A  ERRORERROR'
2018-11-19 21:55:46 [DEBUG   ] Got command RPT from MCS: ref #562626
2018-11-19 21:55:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 21:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562626  77 58441 78946518 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 21:56:06 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:56:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:56:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:56:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:56:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:56:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:56:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:56:24 [DEBUG   ] Finished updating board flags in 1.344 seconds
2018-11-19 21:56:26 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:56:46 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 21:57:06 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 21:57:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 21:57:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:57:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 21:57:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:57:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:57:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:57:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:57:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:57:24 [DEBUG   ] Finished updating board flags in 1.246 seconds
2018-11-19 21:57:26 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:57:46 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:58:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:58:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:58:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 21:58:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:58:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:58:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:58:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:58:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:58:24 [DEBUG   ] Finished updating board flags in 1.151 seconds
2018-11-19 21:58:26 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:58:46 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:59:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:59:08 [DEBUG   ] Got command BAM from MCS: ref #562651
2018-11-19 21:59:08 [INFO    ] Finished processing BAM request
2018-11-19 21:59:08 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 21:59:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562651   8 58441 79148025 A  ERROR'
2018-11-19 21:59:08 [DEBUG   ] Got command BAM from MCS: ref #562650
2018-11-19 21:59:08 [INFO    ] Finished processing BAM request
2018-11-19 21:59:08 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 21:59:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562650   8 58441 79148029 A  ERROR'
2018-11-19 21:59:08 [DEBUG   ] Got command DRX from MCS: ref #562652
2018-11-19 21:59:08 [INFO    ] sending drx command to lwa16
2018-11-19 21:59:08 [DEBUG   ] run_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 1 35100000.0 7 9 9']
2018-11-19 21:59:08 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 21:59:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   562652   8 58441 79148105 A  ERROR'
2018-11-19 21:59:08 [DEBUG   ] Got command DRX from MCS: ref #562653
2018-11-19 21:59:08 [INFO    ] sending drx command to lwa34
2018-11-19 21:59:08 [DEBUG   ] run_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa34', '/lwa/software/drx 1 1 64500000.0 7 9 9']
2018-11-19 21:59:08 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 21:59:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   562653   8 58441 79148109 A  ERROR'
2018-11-19 21:59:08 [DEBUG   ] Got command DRX from MCS: ref #562655
2018-11-19 21:59:08 [INFO    ] sending drx command to lwa34
2018-11-19 21:59:08 [DEBUG   ] run_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa34', '/lwa/software/drx 1 2 79200000.0 7 9 9']
2018-11-19 21:59:08 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 21:59:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   562655   8 58441 79148113 A  ERROR'
2018-11-19 21:59:08 [DEBUG   ] Got command DRX from MCS: ref #562654
2018-11-19 21:59:08 [INFO    ] sending drx command to lwa16
2018-11-19 21:59:08 [DEBUG   ] run_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa16', '/lwa/software/drx 5 2 49800000.0 7 9 9']
2018-11-19 21:59:08 [DEBUG   ] drx = exited with status True, code 0x00
2018-11-19 21:59:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_DRX   562654   8 58441 79148117 A  ERROR'
2018-11-19 21:59:08 [INFO    ] Finished starting DRX
2018-11-19 21:59:08 [INFO    ] Finished starting DRX
2018-11-19 21:59:08 [INFO    ] Finished starting DRX
2018-11-19 21:59:08 [INFO    ] Finished starting DRX
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 21:59:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 21:59:08 [DEBUG   ] spawn_process: at 2018-11-19 21:59:08 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 21:59:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 21:59:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 21:59:24 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-19 21:59:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:59:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:59:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:59:24 [WARNING ] (13, 1, 'bot')
2018-11-19 21:59:24 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 21:59:24 [DEBUG   ] Finished updating board flags in 1.017 seconds
2018-11-19 21:59:26 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:59:46 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 21:59:47 [DEBUG   ] Got command BAM from MCS: ref #562658
2018-11-19 21:59:47 [INFO    ] Finished processing BAM request
2018-11-19 21:59:47 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 21:59:47 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562658   8 58441 79187025 A  ERROR'
2018-11-19 21:59:47 [DEBUG   ] Got command BAM from MCS: ref #562659
2018-11-19 21:59:47 [INFO    ] Finished processing BAM request
2018-11-19 21:59:47 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 21:59:47 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562659   8 58441 79187029 A  ERROR'
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 21:59:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 21:59:47 [DEBUG   ] spawn_process: at 2018-11-19 21:59:47 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:00:01 [DEBUG   ] Got command RPT from MCS: ref #562662
2018-11-19 22:00:01 [DEBUG   ] summary = ERROR
2018-11-19 22:00:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562662  13 58441 79201229 A  ERRORERROR'
2018-11-19 22:00:01 [DEBUG   ] Got command RPT from MCS: ref #562663
2018-11-19 22:00:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:00:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562663  77 58441 79201433 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:00:01 [DEBUG   ] Got command RPT from MCS: ref #562664
2018-11-19 22:00:01 [DEBUG   ] summary = ERROR
2018-11-19 22:00:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562664  13 58441 79201637 A  ERRORERROR'
2018-11-19 22:00:01 [DEBUG   ] Got command RPT from MCS: ref #562666
2018-11-19 22:00:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:00:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562666  77 58441 79201829 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:00:02 [DEBUG   ] Got command RPT from MCS: ref #562667
2018-11-19 22:00:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:00:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562667  12 58441 79202033 A  ERRORB›™š'
2018-11-19 22:00:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:00:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 22:00:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:00:26 [DEBUG   ] Got command BAM from MCS: ref #562672
2018-11-19 22:00:26 [INFO    ] Finished processing BAM request
2018-11-19 22:00:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:00:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562672   8 58441 79226033 A  ERROR'
2018-11-19 22:00:26 [DEBUG   ] Got command BAM from MCS: ref #562673
2018-11-19 22:00:26 [INFO    ] Finished processing BAM request
2018-11-19 22:00:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:00:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562673   8 58441 79226037 A  ERROR'
2018-11-19 22:00:26 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 22:00:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:00:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:00:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:00:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:00:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:00:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:00:26 [DEBUG   ] Finished updating board flags in 2.535 seconds
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:00:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:00:26 [DEBUG   ] spawn_process: at 2018-11-19 22:00:26 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:00:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:00:46 [DEBUG   ] Got command RPT from MCS: ref #562674
2018-11-19 22:00:46 [DEBUG   ] summary = ERROR
2018-11-19 22:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562674  13 58441 79246521 A  ERRORERROR'
2018-11-19 22:00:46 [DEBUG   ] Got command RPT from MCS: ref #562675
2018-11-19 22:00:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562675  77 58441 79246522 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:01:05 [DEBUG   ] Got command BAM from MCS: ref #562691
2018-11-19 22:01:05 [INFO    ] Finished processing BAM request
2018-11-19 22:01:05 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:01:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562691   8 58441 79265029 A  ERROR'
2018-11-19 22:01:05 [DEBUG   ] Got command BAM from MCS: ref #562692
2018-11-19 22:01:05 [INFO    ] Finished processing BAM request
2018-11-19 22:01:05 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:01:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562692   8 58441 79265033 A  ERROR'
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:01:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:01:05 [DEBUG   ] spawn_process: at 2018-11-19 22:01:05 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:01:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:01:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 22:01:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:01:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-19 22:01:25 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:01:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:01:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:01:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:01:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:01:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:01:25 [DEBUG   ] Finished updating board flags in 1.430 seconds
2018-11-19 22:01:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:01:44 [DEBUG   ] Got command BAM from MCS: ref #562697
2018-11-19 22:01:44 [INFO    ] Finished processing BAM request
2018-11-19 22:01:44 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:01:44 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562697   8 58441 79304029 A  ERROR'
2018-11-19 22:01:44 [DEBUG   ] Got command BAM from MCS: ref #562698
2018-11-19 22:01:44 [INFO    ] Finished processing BAM request
2018-11-19 22:01:44 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:01:44 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562698   8 58441 79304033 A  ERROR'
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:01:44 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:01:44 [DEBUG   ] spawn_process: at 2018-11-19 22:01:44 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:01:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:02:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:02:23 [DEBUG   ] Got command BAM from MCS: ref #562700
2018-11-19 22:02:23 [INFO    ] Finished processing BAM request
2018-11-19 22:02:23 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:02:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562700   8 58441 79343029 A  ERROR'
2018-11-19 22:02:23 [DEBUG   ] Got command BAM from MCS: ref #562699
2018-11-19 22:02:23 [INFO    ] Finished processing BAM request
2018-11-19 22:02:23 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:02:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562699   8 58441 79343033 A  ERROR'
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:02:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:02:23 [DEBUG   ] spawn_process: at 2018-11-19 22:02:23 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:02:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:02:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 22:02:25 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:02:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:02:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:02:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:02:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:02:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:02:25 [DEBUG   ] Finished updating board flags in 1.397 seconds
2018-11-19 22:02:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:02:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:03:02 [DEBUG   ] Got command BAM from MCS: ref #562701
2018-11-19 22:03:02 [INFO    ] Finished processing BAM request
2018-11-19 22:03:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:03:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562701   8 58441 79382033 A  ERROR'
2018-11-19 22:03:02 [DEBUG   ] Got command BAM from MCS: ref #562702
2018-11-19 22:03:02 [INFO    ] Finished processing BAM request
2018-11-19 22:03:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:03:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562702   8 58441 79382037 A  ERROR'
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:03:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:03:02 [DEBUG   ] spawn_process: at 2018-11-19 22:03:02 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:03:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:03:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-19 22:03:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:03:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:03:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 22:03:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:03:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:03:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:03:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:03:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:03:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:03:25 [DEBUG   ] Finished updating board flags in 1.247 seconds
2018-11-19 22:03:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:03:41 [DEBUG   ] Got command BAM from MCS: ref #562704
2018-11-19 22:03:41 [INFO    ] Finished processing BAM request
2018-11-19 22:03:41 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:03:41 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562704   8 58441 79421033 A  ERROR'
2018-11-19 22:03:41 [DEBUG   ] Got command BAM from MCS: ref #562703
2018-11-19 22:03:41 [INFO    ] Finished processing BAM request
2018-11-19 22:03:41 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:03:41 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562703   8 58441 79421037 A  ERROR'
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:03:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:03:41 [DEBUG   ] spawn_process: at 2018-11-19 22:03:41 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:03:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:04:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:04:20 [DEBUG   ] Got command BAM from MCS: ref #562709
2018-11-19 22:04:20 [INFO    ] Finished processing BAM request
2018-11-19 22:04:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:04:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562709   8 58441 79460029 A  ERROR'
2018-11-19 22:04:20 [DEBUG   ] Got command BAM from MCS: ref #562710
2018-11-19 22:04:20 [INFO    ] Finished processing BAM request
2018-11-19 22:04:20 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:04:20 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562710   8 58441 79460033 A  ERROR'
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:04:20 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:04:20 [DEBUG   ] spawn_process: at 2018-11-19 22:04:20 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:04:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:04:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 22:04:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:04:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:04:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:04:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:04:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:04:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:04:26 [DEBUG   ] Finished updating board flags in 1.964 seconds
2018-11-19 22:04:26 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-19 22:04:46 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 22:04:59 [DEBUG   ] Got command BAM from MCS: ref #562711
2018-11-19 22:04:59 [INFO    ] Finished processing BAM request
2018-11-19 22:04:59 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:04:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562711   8 58441 79499033 A  ERROR'
2018-11-19 22:04:59 [DEBUG   ] Got command BAM from MCS: ref #562712
2018-11-19 22:04:59 [INFO    ] Finished processing BAM request
2018-11-19 22:04:59 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:04:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562712   8 58441 79499037 A  ERROR'
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:04:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:04:59 [DEBUG   ] spawn_process: at 2018-11-19 22:04:59 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:05:01 [DEBUG   ] Got command RPT from MCS: ref #562713
2018-11-19 22:05:01 [DEBUG   ] summary = ERROR
2018-11-19 22:05:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562713  13 58441 79501317 A  ERRORERROR'
2018-11-19 22:05:01 [DEBUG   ] Got command RPT from MCS: ref #562714
2018-11-19 22:05:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:05:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562714  77 58441 79501521 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:05:01 [DEBUG   ] Got command RPT from MCS: ref #562715
2018-11-19 22:05:01 [DEBUG   ] summary = ERROR
2018-11-19 22:05:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562715  13 58441 79501713 A  ERRORERROR'
2018-11-19 22:05:01 [DEBUG   ] Got command RPT from MCS: ref #562716
2018-11-19 22:05:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:05:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562716  77 58441 79501917 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:05:02 [DEBUG   ] Got command RPT from MCS: ref #562717
2018-11-19 22:05:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:05:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562717  12 58441 79502121 A  ERRORB¢ff'
2018-11-19 22:05:06 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:05:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:05:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:05:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:05:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:05:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:05:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:05:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:05:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:05:25 [DEBUG   ] Finished updating board flags in 0.997 seconds
2018-11-19 22:05:26 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:05:37 [DEBUG   ] Got command BAM from MCS: ref #562718
2018-11-19 22:05:37 [INFO    ] Finished processing BAM request
2018-11-19 22:05:37 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:05:37 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562718   8 58441 79537033 A  ERROR'
2018-11-19 22:05:37 [DEBUG   ] Got command BAM from MCS: ref #562719
2018-11-19 22:05:37 [INFO    ] Finished processing BAM request
2018-11-19 22:05:37 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:05:37 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562719   8 58441 79537037 A  ERROR'
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:05:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:05:37 [DEBUG   ] spawn_process: at 2018-11-19 22:05:37 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:05:46 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:05:46 [DEBUG   ] Got command RPT from MCS: ref #562720
2018-11-19 22:05:46 [DEBUG   ] summary = ERROR
2018-11-19 22:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562720  13 58441 79546513 A  ERRORERROR'
2018-11-19 22:05:46 [DEBUG   ] Got command RPT from MCS: ref #562721
2018-11-19 22:05:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562721  77 58441 79546514 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:06:06 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 22:06:15 [DEBUG   ] Got command BAM from MCS: ref #562738
2018-11-19 22:06:15 [INFO    ] Finished processing BAM request
2018-11-19 22:06:15 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:06:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562738   8 58441 79575025 A  ERROR'
2018-11-19 22:06:15 [DEBUG   ] Got command BAM from MCS: ref #562737
2018-11-19 22:06:15 [INFO    ] Finished processing BAM request
2018-11-19 22:06:15 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:06:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562737   8 58441 79575029 A  ERROR'
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:06:15 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:06:15 [DEBUG   ] spawn_process: at 2018-11-19 22:06:15 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:06:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:06:26 [DEBUG   ] Finished updating ant stats in 0.430 seconds
2018-11-19 22:06:26 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:06:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:06:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:06:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:06:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:06:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:06:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:06:26 [DEBUG   ] Finished updating board flags in 1.856 seconds
2018-11-19 22:06:46 [DEBUG   ] Finished updating ant stats in 0.375 seconds
2018-11-19 22:06:53 [DEBUG   ] Got command BAM from MCS: ref #562740
2018-11-19 22:06:53 [INFO    ] Finished processing BAM request
2018-11-19 22:06:53 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:06:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562740   8 58441 79613025 A  ERROR'
2018-11-19 22:06:53 [DEBUG   ] Got command BAM from MCS: ref #562739
2018-11-19 22:06:53 [INFO    ] Finished processing BAM request
2018-11-19 22:06:53 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:06:53 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562739   8 58441 79613029 A  ERROR'
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:06:53 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:06:53 [DEBUG   ] spawn_process: at 2018-11-19 22:06:53 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:07:07 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:07:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 22:07:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:07:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:07:25 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:07:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:07:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:07:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:07:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:07:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:07:25 [DEBUG   ] Finished updating board flags in 1.212 seconds
2018-11-19 22:07:27 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 22:07:31 [DEBUG   ] Got command BAM from MCS: ref #562745
2018-11-19 22:07:31 [INFO    ] Finished processing BAM request
2018-11-19 22:07:31 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:07:31 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562745   8 58441 79651033 A  ERROR'
2018-11-19 22:07:31 [DEBUG   ] Got command BAM from MCS: ref #562746
2018-11-19 22:07:31 [INFO    ] Finished processing BAM request
2018-11-19 22:07:31 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:07:31 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562746   8 58441 79651037 A  ERROR'
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:07:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:07:31 [DEBUG   ] spawn_process: at 2018-11-19 22:07:31 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:07:47 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:08:07 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:08:09 [DEBUG   ] Got command BAM from MCS: ref #562748
2018-11-19 22:08:09 [INFO    ] Finished processing BAM request
2018-11-19 22:08:09 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:08:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562748   8 58441 79689033 A  ERROR'
2018-11-19 22:08:09 [DEBUG   ] Got command BAM from MCS: ref #562747
2018-11-19 22:08:09 [INFO    ] Finished processing BAM request
2018-11-19 22:08:09 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:08:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562747   8 58441 79689037 A  ERROR'
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:08:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:08:09 [DEBUG   ] spawn_process: at 2018-11-19 22:08:09 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:08:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:08:25 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:08:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:08:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:08:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:08:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:08:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:08:25 [DEBUG   ] Finished updating board flags in 1.180 seconds
2018-11-19 22:08:27 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:08:47 [DEBUG   ] Got command BAM from MCS: ref #562750
2018-11-19 22:08:47 [INFO    ] Finished processing BAM request
2018-11-19 22:08:47 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:08:47 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562750   8 58441 79727033 A  ERROR'
2018-11-19 22:08:47 [DEBUG   ] Got command BAM from MCS: ref #562749
2018-11-19 22:08:47 [INFO    ] Finished processing BAM request
2018-11-19 22:08:47 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:08:47 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562749   8 58441 79727037 A  ERROR'
2018-11-19 22:08:47 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:08:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:08:47 [DEBUG   ] spawn_process: at 2018-11-19 22:08:47 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:09:07 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:09:25 [DEBUG   ] Got command BAM from MCS: ref #562752
2018-11-19 22:09:25 [INFO    ] Finished processing BAM request
2018-11-19 22:09:25 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:09:25 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562752   8 58441 79765037 A  ERROR'
2018-11-19 22:09:25 [DEBUG   ] Got command BAM from MCS: ref #562751
2018-11-19 22:09:25 [INFO    ] Finished processing BAM request
2018-11-19 22:09:25 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:09:25 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562751   8 58441 79765041 A  ERROR'
2018-11-19 22:09:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 22:09:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 22:09:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:09:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:09:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:09:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:09:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:09:25 [WARNING ] (13, 1, 'bot')
2018-11-19 22:09:25 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:09:25 [DEBUG   ] Finished updating board flags in 0.989 seconds
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:09:25 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:09:25 [DEBUG   ] spawn_process: at 2018-11-19 22:09:25 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:09:27 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:09:47 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:10:01 [DEBUG   ] Got command RPT from MCS: ref #562755
2018-11-19 22:10:01 [DEBUG   ] summary = ERROR
2018-11-19 22:10:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562755  13 58441 79801401 A  ERRORERROR'
2018-11-19 22:10:01 [DEBUG   ] Got command RPT from MCS: ref #562756
2018-11-19 22:10:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:10:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562756  77 58441 79801605 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:10:01 [DEBUG   ] Got command RPT from MCS: ref #562757
2018-11-19 22:10:01 [DEBUG   ] summary = ERROR
2018-11-19 22:10:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562757  13 58441 79801809 A  ERRORERROR'
2018-11-19 22:10:02 [DEBUG   ] Got command RPT from MCS: ref #562759
2018-11-19 22:10:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:10:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562759  77 58441 79802001 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:10:02 [DEBUG   ] Got command RPT from MCS: ref #562762
2018-11-19 22:10:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:10:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562762  12 58441 79802205 A  ERRORB¢ff'
2018-11-19 22:10:03 [DEBUG   ] Got command BAM from MCS: ref #562760
2018-11-19 22:10:03 [INFO    ] Finished processing BAM request
2018-11-19 22:10:03 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:10:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562760   8 58441 79803025 A  ERROR'
2018-11-19 22:10:03 [DEBUG   ] Got command BAM from MCS: ref #562761
2018-11-19 22:10:03 [INFO    ] Finished processing BAM request
2018-11-19 22:10:03 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:10:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562761   8 58441 79803029 A  ERROR'
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:10:03 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:10:03 [DEBUG   ] spawn_process: at 2018-11-19 22:10:03 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:10:07 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 22:10:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:10:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 22:10:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:10:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 22:10:27 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:10:27 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 22:10:27 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:10:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:10:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:10:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:10:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:10:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:10:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:10:27 [DEBUG   ] Finished updating board flags in 2.330 seconds
2018-11-19 22:10:41 [DEBUG   ] Got command BAM from MCS: ref #562772
2018-11-19 22:10:41 [INFO    ] Finished processing BAM request
2018-11-19 22:10:41 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:10:41 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562772   8 58441 79841025 A  ERROR'
2018-11-19 22:10:41 [DEBUG   ] Got command BAM from MCS: ref #562771
2018-11-19 22:10:41 [INFO    ] Finished processing BAM request
2018-11-19 22:10:41 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:10:41 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562771   8 58441 79841029 A  ERROR'
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:10:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:10:41 [DEBUG   ] spawn_process: at 2018-11-19 22:10:41 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:10:46 [DEBUG   ] Got command RPT from MCS: ref #562773
2018-11-19 22:10:46 [DEBUG   ] summary = ERROR
2018-11-19 22:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562773  13 58441 79846521 A  ERRORERROR'
2018-11-19 22:10:46 [DEBUG   ] Got command RPT from MCS: ref #562774
2018-11-19 22:10:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562774  77 58441 79846522 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:10:47 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:11:07 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 22:11:19 [DEBUG   ] Got command BAM from MCS: ref #562791
2018-11-19 22:11:19 [INFO    ] Finished processing BAM request
2018-11-19 22:11:19 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:11:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562791   8 58441 79879029 A  ERROR'
2018-11-19 22:11:19 [DEBUG   ] Got command BAM from MCS: ref #562790
2018-11-19 22:11:19 [INFO    ] Finished processing BAM request
2018-11-19 22:11:19 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:11:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562790   8 58441 79879033 A  ERROR'
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:11:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:11:19 [DEBUG   ] spawn_process: at 2018-11-19 22:11:19 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:11:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:11:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:11:26 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:11:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:11:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:11:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:11:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:11:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:11:26 [DEBUG   ] Finished updating board flags in 1.248 seconds
2018-11-19 22:11:27 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:11:47 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:11:57 [DEBUG   ] Got command BAM from MCS: ref #562793
2018-11-19 22:11:57 [INFO    ] Finished processing BAM request
2018-11-19 22:11:57 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:11:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562793   8 58441 79917029 A  ERROR'
2018-11-19 22:11:57 [DEBUG   ] Got command BAM from MCS: ref #562792
2018-11-19 22:11:57 [INFO    ] Finished processing BAM request
2018-11-19 22:11:57 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:11:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562792   8 58441 79917033 A  ERROR'
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:11:57 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:11:57 [DEBUG   ] spawn_process: at 2018-11-19 22:11:57 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:12:07 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 22:12:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 22:12:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 22:12:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:12:26 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:12:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:12:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:12:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:12:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:12:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:12:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:12:26 [DEBUG   ] Finished updating board flags in 1.279 seconds
2018-11-19 22:12:27 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 22:12:35 [DEBUG   ] Got command BAM from MCS: ref #562795
2018-11-19 22:12:35 [INFO    ] Finished processing BAM request
2018-11-19 22:12:35 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:12:35 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562795   8 58441 79955033 A  ERROR'
2018-11-19 22:12:35 [DEBUG   ] Got command BAM from MCS: ref #562794
2018-11-19 22:12:35 [INFO    ] Finished processing BAM request
2018-11-19 22:12:35 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:12:35 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562794   8 58441 79955037 A  ERROR'
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:12:35 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:12:35 [DEBUG   ] spawn_process: at 2018-11-19 22:12:35 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:12:47 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 22:13:07 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:13:13 [DEBUG   ] Got command BAM from MCS: ref #562800
2018-11-19 22:13:13 [INFO    ] Finished processing BAM request
2018-11-19 22:13:13 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:13:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562800   8 58441 79993029 A  ERROR'
2018-11-19 22:13:13 [DEBUG   ] Got command BAM from MCS: ref #562801
2018-11-19 22:13:13 [INFO    ] Finished processing BAM request
2018-11-19 22:13:13 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:13:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562801   8 58441 79993033 A  ERROR'
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:13:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:13:13 [DEBUG   ] spawn_process: at 2018-11-19 22:13:13 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:13:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 22:13:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:13:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:13:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 22:13:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-19 22:13:26 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:13:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:13:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:13:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:13:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:13:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:13:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:13:26 [DEBUG   ] Finished updating board flags in 1.121 seconds
2018-11-19 22:13:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:13:47 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:13:51 [DEBUG   ] Got command BAM from MCS: ref #562803
2018-11-19 22:13:51 [INFO    ] Finished processing BAM request
2018-11-19 22:13:51 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:13:51 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562803   8 58441 80031025 A  ERROR'
2018-11-19 22:13:51 [DEBUG   ] Got command BAM from MCS: ref #562802
2018-11-19 22:13:51 [INFO    ] Finished processing BAM request
2018-11-19 22:13:51 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:13:51 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562802   8 58441 80031029 A  ERROR'
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:13:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:13:51 [DEBUG   ] spawn_process: at 2018-11-19 22:13:51 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:14:07 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:14:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:14:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:14:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 22:14:26 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:14:26 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:14:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:14:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:14:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:14:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:14:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:14:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:14:26 [DEBUG   ] Finished updating board flags in 1.005 seconds
2018-11-19 22:14:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:14:29 [DEBUG   ] Got command BAM from MCS: ref #562804
2018-11-19 22:14:29 [INFO    ] Finished processing BAM request
2018-11-19 22:14:29 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:14:29 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562804   8 58441 80069029 A  ERROR'
2018-11-19 22:14:29 [DEBUG   ] Got command BAM from MCS: ref #562805
2018-11-19 22:14:29 [INFO    ] Finished processing BAM request
2018-11-19 22:14:29 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:14:29 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562805   8 58441 80069033 A  ERROR'
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:14:29 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:14:29 [DEBUG   ] spawn_process: at 2018-11-19 22:14:29 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:14:47 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 22:15:01 [DEBUG   ] Got command RPT from MCS: ref #562806
2018-11-19 22:15:01 [DEBUG   ] summary = ERROR
2018-11-19 22:15:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562806  13 58441 80101485 A  ERRORERROR'
2018-11-19 22:15:01 [DEBUG   ] Got command RPT from MCS: ref #562807
2018-11-19 22:15:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:15:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562807  77 58441 80101689 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:15:01 [DEBUG   ] Got command RPT from MCS: ref #562808
2018-11-19 22:15:01 [DEBUG   ] summary = ERROR
2018-11-19 22:15:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562808  13 58441 80101893 A  ERRORERROR'
2018-11-19 22:15:02 [DEBUG   ] Got command RPT from MCS: ref #562809
2018-11-19 22:15:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:15:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562809  77 58441 80102085 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:15:02 [DEBUG   ] Got command RPT from MCS: ref #562810
2018-11-19 22:15:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:15:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562810  12 58441 80102289 A  ERRORB£ff'
2018-11-19 22:15:07 [DEBUG   ] Got command BAM from MCS: ref #562812
2018-11-19 22:15:07 [INFO    ] Finished processing BAM request
2018-11-19 22:15:07 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:15:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562812   8 58441 80107037 A  ERROR'
2018-11-19 22:15:07 [DEBUG   ] Got command BAM from MCS: ref #562811
2018-11-19 22:15:07 [INFO    ] Finished processing BAM request
2018-11-19 22:15:07 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:15:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562811   8 58441 80107045 A  ERROR'
2018-11-19 22:15:07 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:15:07 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:15:07 [DEBUG   ] spawn_process: at 2018-11-19 22:15:07 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:15:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 22:15:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:15:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 22:15:26 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:15:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:15:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:15:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:15:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:15:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:15:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:15:26 [DEBUG   ] Finished updating board flags in 0.980 seconds
2018-11-19 22:15:27 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 22:15:45 [DEBUG   ] Got command BAM from MCS: ref #562814
2018-11-19 22:15:45 [INFO    ] Finished processing BAM request
2018-11-19 22:15:45 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:15:45 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562814   8 58441 80145037 A  ERROR'
2018-11-19 22:15:45 [DEBUG   ] Got command BAM from MCS: ref #562813
2018-11-19 22:15:45 [INFO    ] Finished processing BAM request
2018-11-19 22:15:45 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:15:45 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562813   8 58441 80145045 A  ERROR'
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:15:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:15:45 [DEBUG   ] spawn_process: at 2018-11-19 22:15:45 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:15:46 [DEBUG   ] Got command RPT from MCS: ref #562815
2018-11-19 22:15:46 [DEBUG   ] summary = ERROR
2018-11-19 22:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562815  13 58441 80146521 A  ERRORERROR'
2018-11-19 22:15:46 [DEBUG   ] Got command RPT from MCS: ref #562816
2018-11-19 22:15:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562816  77 58441 80146522 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:15:47 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:16:07 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:16:23 [DEBUG   ] Got command BAM from MCS: ref #562837
2018-11-19 22:16:23 [INFO    ] Finished processing BAM request
2018-11-19 22:16:23 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:16:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562837   8 58441 80183029 A  ERROR'
2018-11-19 22:16:23 [DEBUG   ] Got command BAM from MCS: ref #562836
2018-11-19 22:16:23 [INFO    ] Finished processing BAM request
2018-11-19 22:16:23 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:16:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562836   8 58441 80183033 A  ERROR'
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:16:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:16:23 [DEBUG   ] spawn_process: at 2018-11-19 22:16:23 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:16:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:16:26 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:16:26 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:16:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:16:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:16:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:16:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:16:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:16:26 [DEBUG   ] Finished updating board flags in 1.271 seconds
2018-11-19 22:16:27 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:16:47 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:17:01 [DEBUG   ] Got command BAM from MCS: ref #562839
2018-11-19 22:17:01 [INFO    ] Finished processing BAM request
2018-11-19 22:17:01 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:17:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562839   8 58441 80221033 A  ERROR'
2018-11-19 22:17:01 [DEBUG   ] Got command BAM from MCS: ref #562838
2018-11-19 22:17:01 [INFO    ] Finished processing BAM request
2018-11-19 22:17:01 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:17:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562838   8 58441 80221037 A  ERROR'
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:17:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:17:01 [DEBUG   ] spawn_process: at 2018-11-19 22:17:01 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:17:07 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:17:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 22:17:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:17:26 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:17:26 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:17:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:17:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:17:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:17:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:17:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:17:26 [DEBUG   ] Finished updating board flags in 1.252 seconds
2018-11-19 22:17:27 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:17:39 [DEBUG   ] Got command BAM from MCS: ref #562841
2018-11-19 22:17:39 [INFO    ] Finished processing BAM request
2018-11-19 22:17:39 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:17:39 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562841   8 58441 80259025 A  ERROR'
2018-11-19 22:17:39 [DEBUG   ] Got command BAM from MCS: ref #562840
2018-11-19 22:17:39 [INFO    ] Finished processing BAM request
2018-11-19 22:17:39 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:17:39 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562840   8 58441 80259029 A  ERROR'
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:17:39 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:17:39 [DEBUG   ] spawn_process: at 2018-11-19 22:17:39 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:17:47 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:18:07 [DEBUG   ] Finished updating ant stats in 0.398 seconds
2018-11-19 22:18:17 [DEBUG   ] Got command BAM from MCS: ref #562843
2018-11-19 22:18:17 [INFO    ] Finished processing BAM request
2018-11-19 22:18:17 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:18:17 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562843   8 58441 80297029 A  ERROR'
2018-11-19 22:18:17 [DEBUG   ] Got command BAM from MCS: ref #562842
2018-11-19 22:18:17 [INFO    ] Finished processing BAM request
2018-11-19 22:18:17 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:18:17 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562842   8 58441 80297033 A  ERROR'
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:18:17 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:18:17 [DEBUG   ] spawn_process: at 2018-11-19 22:18:17 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:18:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:18:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:18:26 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:18:26 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:18:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:18:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:18:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:18:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:18:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:18:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:18:26 [DEBUG   ] Finished updating board flags in 1.131 seconds
2018-11-19 22:18:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:18:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:18:54 [DEBUG   ] Got command BAM from MCS: ref #562844
2018-11-19 22:18:54 [INFO    ] Finished processing BAM request
2018-11-19 22:18:54 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:18:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562844   8 58441 80334033 A  ERROR'
2018-11-19 22:18:54 [DEBUG   ] Got command BAM from MCS: ref #562845
2018-11-19 22:18:54 [INFO    ] Finished processing BAM request
2018-11-19 22:18:54 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:18:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562845   8 58441 80334037 A  ERROR'
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:18:54 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:18:54 [DEBUG   ] spawn_process: at 2018-11-19 22:18:54 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:19:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:19:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:19:27 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:19:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:19:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:19:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:19:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:19:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:19:27 [DEBUG   ] Finished updating board flags in 1.617 seconds
2018-11-19 22:19:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:19:31 [DEBUG   ] Got command BAM from MCS: ref #562851
2018-11-19 22:19:31 [INFO    ] Finished processing BAM request
2018-11-19 22:19:31 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:19:31 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562851   8 58441 80371033 A  ERROR'
2018-11-19 22:19:31 [DEBUG   ] Got command BAM from MCS: ref #562850
2018-11-19 22:19:31 [INFO    ] Finished processing BAM request
2018-11-19 22:19:31 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:19:31 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562850   8 58441 80371037 A  ERROR'
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:19:31 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:19:31 [DEBUG   ] spawn_process: at 2018-11-19 22:19:31 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:19:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:20:01 [DEBUG   ] Got command RPT from MCS: ref #562854
2018-11-19 22:20:01 [DEBUG   ] summary = ERROR
2018-11-19 22:20:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562854  13 58441 80401569 A  ERRORERROR'
2018-11-19 22:20:01 [DEBUG   ] Got command RPT from MCS: ref #562855
2018-11-19 22:20:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:20:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562855  77 58441 80401773 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:20:01 [DEBUG   ] Got command RPT from MCS: ref #562856
2018-11-19 22:20:01 [DEBUG   ] summary = ERROR
2018-11-19 22:20:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562856  13 58441 80401977 A  ERRORERROR'
2018-11-19 22:20:02 [DEBUG   ] Got command RPT from MCS: ref #562858
2018-11-19 22:20:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:20:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562858  77 58441 80402169 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:20:02 [DEBUG   ] Got command RPT from MCS: ref #562859
2018-11-19 22:20:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:20:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562859  12 58441 80402373 A  ERRORB£ff'
2018-11-19 22:20:08 [DEBUG   ] Got command BAM from MCS: ref #562864
2018-11-19 22:20:08 [INFO    ] Finished processing BAM request
2018-11-19 22:20:08 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:20:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562864   8 58441 80408033 A  ERROR'
2018-11-19 22:20:08 [DEBUG   ] Got command BAM from MCS: ref #562865
2018-11-19 22:20:08 [INFO    ] Finished processing BAM request
2018-11-19 22:20:08 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:20:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562865   8 58441 80408037 A  ERROR'
2018-11-19 22:20:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:20:08 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:20:08 [DEBUG   ] spawn_process: at 2018-11-19 22:20:08 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:20:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 22:20:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:20:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:20:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 22:20:26 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:20:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:20:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:20:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:20:26 [WARNING ] (13, 1, 'bot')
2018-11-19 22:20:26 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:20:26 [DEBUG   ] Finished updating board flags in 0.993 seconds
2018-11-19 22:20:28 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 22:20:45 [DEBUG   ] Got command BAM from MCS: ref #562867
2018-11-19 22:20:45 [INFO    ] Finished processing BAM request
2018-11-19 22:20:45 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:20:45 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562867   8 58441 80445029 A  ERROR'
2018-11-19 22:20:45 [DEBUG   ] Got command BAM from MCS: ref #562866
2018-11-19 22:20:45 [INFO    ] Finished processing BAM request
2018-11-19 22:20:45 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:20:45 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562866   8 58441 80445033 A  ERROR'
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:20:45 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:20:45 [DEBUG   ] spawn_process: at 2018-11-19 22:20:45 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:20:46 [DEBUG   ] Got command RPT from MCS: ref #562868
2018-11-19 22:20:46 [DEBUG   ] summary = ERROR
2018-11-19 22:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562868  13 58441 80446521 A  ERRORERROR'
2018-11-19 22:20:46 [DEBUG   ] Got command RPT from MCS: ref #562869
2018-11-19 22:20:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562869  77 58441 80446522 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:20:48 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:21:08 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:21:22 [DEBUG   ] Got command BAM from MCS: ref #562886
2018-11-19 22:21:22 [INFO    ] Finished processing BAM request
2018-11-19 22:21:22 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:21:22 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562886   8 58441 80482033 A  ERROR'
2018-11-19 22:21:22 [DEBUG   ] Got command BAM from MCS: ref #562885
2018-11-19 22:21:22 [INFO    ] Finished processing BAM request
2018-11-19 22:21:22 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:21:22 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562885   8 58441 80482037 A  ERROR'
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:21:22 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:21:22 [DEBUG   ] spawn_process: at 2018-11-19 22:21:22 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:21:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 22:21:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:21:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:21:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 22:21:27 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:21:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:21:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:21:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:21:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:21:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:21:27 [DEBUG   ] Finished updating board flags in 1.368 seconds
2018-11-19 22:21:28 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:21:48 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 22:21:59 [DEBUG   ] Got command BAM from MCS: ref #562888
2018-11-19 22:21:59 [INFO    ] Finished processing BAM request
2018-11-19 22:21:59 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:21:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562888   8 58441 80519029 A  ERROR'
2018-11-19 22:21:59 [DEBUG   ] Got command BAM from MCS: ref #562887
2018-11-19 22:21:59 [INFO    ] Finished processing BAM request
2018-11-19 22:21:59 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:21:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562887   8 58441 80519033 A  ERROR'
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:21:59 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:21:59 [DEBUG   ] spawn_process: at 2018-11-19 22:21:59 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:22:08 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:22:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:22:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 22:22:27 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:22:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:22:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:22:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:22:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:22:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:22:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:22:27 [DEBUG   ] Finished updating board flags in 1.259 seconds
2018-11-19 22:22:28 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:22:36 [DEBUG   ] Got command BAM from MCS: ref #562893
2018-11-19 22:22:36 [INFO    ] Finished processing BAM request
2018-11-19 22:22:36 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:22:36 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562893   8 58441 80556033 A  ERROR'
2018-11-19 22:22:36 [DEBUG   ] Got command BAM from MCS: ref #562894
2018-11-19 22:22:36 [INFO    ] Finished processing BAM request
2018-11-19 22:22:36 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:22:36 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562894   8 58441 80556037 A  ERROR'
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:22:36 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:22:36 [DEBUG   ] spawn_process: at 2018-11-19 22:22:36 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:22:48 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:23:08 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:23:13 [DEBUG   ] Got command BAM from MCS: ref #562896
2018-11-19 22:23:13 [INFO    ] Finished processing BAM request
2018-11-19 22:23:13 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:23:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562896   8 58441 80593033 A  ERROR'
2018-11-19 22:23:13 [DEBUG   ] Got command BAM from MCS: ref #562895
2018-11-19 22:23:13 [INFO    ] Finished processing BAM request
2018-11-19 22:23:13 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:23:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562895   8 58441 80593037 A  ERROR'
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:23:13 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:23:13 [DEBUG   ] spawn_process: at 2018-11-19 22:23:13 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:23:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 22:23:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:23:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:23:27 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:23:28 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-19 22:23:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:23:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:23:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:23:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:23:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:23:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:23:28 [DEBUG   ] Finished updating board flags in 2.074 seconds
2018-11-19 22:23:48 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 22:23:50 [DEBUG   ] Got command BAM from MCS: ref #562897
2018-11-19 22:23:50 [INFO    ] Finished processing BAM request
2018-11-19 22:23:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:23:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562897   8 58441 80630025 A  ERROR'
2018-11-19 22:23:50 [DEBUG   ] Got command BAM from MCS: ref #562898
2018-11-19 22:23:50 [INFO    ] Finished processing BAM request
2018-11-19 22:23:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:23:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562898   8 58441 80630029 A  ERROR'
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:23:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:23:50 [DEBUG   ] spawn_process: at 2018-11-19 22:23:50 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:24:08 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:24:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 22:24:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:24:27 [DEBUG   ] Got command BAM from MCS: ref #562900
2018-11-19 22:24:27 [INFO    ] Finished processing BAM request
2018-11-19 22:24:27 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:24:27 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562900   8 58441 80667025 A  ERROR'
2018-11-19 22:24:27 [DEBUG   ] Got command BAM from MCS: ref #562899
2018-11-19 22:24:27 [INFO    ] Finished processing BAM request
2018-11-19 22:24:27 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:24:27 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562899   8 58441 80667029 A  ERROR'
2018-11-19 22:24:27 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:24:27 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:24:27 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:24:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:24:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:24:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:24:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:24:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:24:27 [DEBUG   ] Finished updating board flags in 1.109 seconds
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:24:27 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:24:27 [DEBUG   ] spawn_process: at 2018-11-19 22:24:27 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:24:28 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:24:48 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:25:01 [DEBUG   ] Got command RPT from MCS: ref #562901
2018-11-19 22:25:01 [DEBUG   ] summary = ERROR
2018-11-19 22:25:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562901  13 58441 80701661 A  ERRORERROR'
2018-11-19 22:25:01 [DEBUG   ] Got command RPT from MCS: ref #562902
2018-11-19 22:25:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:25:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562902  77 58441 80701853 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:25:02 [DEBUG   ] Got command RPT from MCS: ref #562903
2018-11-19 22:25:02 [DEBUG   ] summary = ERROR
2018-11-19 22:25:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562903  13 58441 80702057 A  ERRORERROR'
2018-11-19 22:25:02 [DEBUG   ] Got command RPT from MCS: ref #562904
2018-11-19 22:25:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:25:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562904  77 58441 80702261 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:25:02 [DEBUG   ] Got command RPT from MCS: ref #562905
2018-11-19 22:25:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:25:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562905  12 58441 80702465 A  ERRORB£ff'
2018-11-19 22:25:04 [DEBUG   ] Got command BAM from MCS: ref #562906
2018-11-19 22:25:04 [INFO    ] Finished processing BAM request
2018-11-19 22:25:04 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:25:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562906   8 58441 80704029 A  ERROR'
2018-11-19 22:25:04 [DEBUG   ] Got command BAM from MCS: ref #562907
2018-11-19 22:25:04 [INFO    ] Finished processing BAM request
2018-11-19 22:25:04 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:25:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562907   8 58441 80704033 A  ERROR'
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:25:04 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:25:04 [DEBUG   ] spawn_process: at 2018-11-19 22:25:04 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:25:08 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:25:27 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:25:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:25:28 [DEBUG   ] Finished updating ant stats in 0.824 seconds
2018-11-19 22:25:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:25:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:25:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:25:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:25:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:25:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:25:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:25:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:25:28 [DEBUG   ] Finished updating board flags in 1.982 seconds
2018-11-19 22:25:41 [DEBUG   ] Got command BAM from MCS: ref #562912
2018-11-19 22:25:41 [INFO    ] Finished processing BAM request
2018-11-19 22:25:41 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:25:41 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562912   8 58441 80741033 A  ERROR'
2018-11-19 22:25:41 [DEBUG   ] Got command BAM from MCS: ref #562913
2018-11-19 22:25:41 [INFO    ] Finished processing BAM request
2018-11-19 22:25:41 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:25:41 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562913   8 58441 80741037 A  ERROR'
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:25:41 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:25:41 [DEBUG   ] spawn_process: at 2018-11-19 22:25:41 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:25:46 [DEBUG   ] Got command RPT from MCS: ref #562914
2018-11-19 22:25:46 [DEBUG   ] summary = ERROR
2018-11-19 22:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562914  13 58441 80746521 A  ERRORERROR'
2018-11-19 22:25:46 [DEBUG   ] Got command RPT from MCS: ref #562915
2018-11-19 22:25:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562915  77 58441 80746522 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:25:48 [DEBUG   ] Finished updating ant stats in 0.774 seconds
2018-11-19 22:26:08 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 22:26:18 [DEBUG   ] Got command BAM from MCS: ref #562932
2018-11-19 22:26:18 [INFO    ] Finished processing BAM request
2018-11-19 22:26:18 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:26:18 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562932   8 58441 80778025 A  ERROR'
2018-11-19 22:26:18 [DEBUG   ] Got command BAM from MCS: ref #562931
2018-11-19 22:26:18 [INFO    ] Finished processing BAM request
2018-11-19 22:26:18 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:26:18 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562931   8 58441 80778029 A  ERROR'
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:26:18 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:26:18 [DEBUG   ] spawn_process: at 2018-11-19 22:26:18 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:26:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:26:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:26:28 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-19 22:26:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:26:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:26:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:26:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:26:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:26:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:26:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:26:28 [DEBUG   ] Finished updating board flags in 1.765 seconds
2018-11-19 22:26:48 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-19 22:26:55 [DEBUG   ] Got command BAM from MCS: ref #562934
2018-11-19 22:26:55 [INFO    ] Finished processing BAM request
2018-11-19 22:26:55 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:26:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562934   8 58441 80815029 A  ERROR'
2018-11-19 22:26:55 [DEBUG   ] Got command BAM from MCS: ref #562933
2018-11-19 22:26:55 [INFO    ] Finished processing BAM request
2018-11-19 22:26:55 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:26:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562933   8 58441 80815033 A  ERROR'
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:26:55 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:26:55 [DEBUG   ] spawn_process: at 2018-11-19 22:26:55 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:27:08 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 22:27:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 22:27:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:27:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:27:28 [DEBUG   ] Finished updating ant stats in 0.430 seconds
2018-11-19 22:27:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:27:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:27:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:27:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:27:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:27:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:27:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:27:28 [DEBUG   ] Finished updating board flags in 1.668 seconds
2018-11-19 22:27:32 [DEBUG   ] Got command BAM from MCS: ref #562936
2018-11-19 22:27:32 [INFO    ] Finished processing BAM request
2018-11-19 22:27:32 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:27:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562936   8 58441 80852033 A  ERROR'
2018-11-19 22:27:32 [DEBUG   ] Got command BAM from MCS: ref #562935
2018-11-19 22:27:32 [INFO    ] Finished processing BAM request
2018-11-19 22:27:32 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:27:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562935   8 58441 80852037 A  ERROR'
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:27:32 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:27:32 [DEBUG   ] spawn_process: at 2018-11-19 22:27:32 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:27:48 [DEBUG   ] Finished updating ant stats in 0.374 seconds
2018-11-19 22:28:09 [DEBUG   ] Got command BAM from MCS: ref #562939
2018-11-19 22:28:09 [INFO    ] Finished processing BAM request
2018-11-19 22:28:09 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:28:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562939   8 58441 80889033 A  ERROR'
2018-11-19 22:28:09 [DEBUG   ] Got command BAM from MCS: ref #562938
2018-11-19 22:28:09 [INFO    ] Finished processing BAM request
2018-11-19 22:28:09 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:28:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562938   8 58441 80889037 A  ERROR'
2018-11-19 22:28:09 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:28:09 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:28:09 [DEBUG   ] spawn_process: at 2018-11-19 22:28:09 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:28:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 22:28:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:28:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:28:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 22:28:27 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:28:27 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:28:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:28:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:28:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:28:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:28:27 [WARNING ] (13, 1, 'bot')
2018-11-19 22:28:27 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:28:27 [DEBUG   ] Finished updating board flags in 0.973 seconds
2018-11-19 22:28:29 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:28:46 [DEBUG   ] Got command BAM from MCS: ref #562944
2018-11-19 22:28:46 [INFO    ] Finished processing BAM request
2018-11-19 22:28:46 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:28:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562944   8 58441 80926025 A  ERROR'
2018-11-19 22:28:46 [DEBUG   ] Got command BAM from MCS: ref #562943
2018-11-19 22:28:46 [INFO    ] Finished processing BAM request
2018-11-19 22:28:46 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:28:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562943   8 58441 80926029 A  ERROR'
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:28:46 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:28:46 [DEBUG   ] spawn_process: at 2018-11-19 22:28:46 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:28:49 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:29:09 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:29:23 [DEBUG   ] Got command BAM from MCS: ref #562946
2018-11-19 22:29:23 [INFO    ] Finished processing BAM request
2018-11-19 22:29:23 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:29:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562946   8 58441 80963025 A  ERROR'
2018-11-19 22:29:23 [DEBUG   ] Got command BAM from MCS: ref #562945
2018-11-19 22:29:23 [INFO    ] Finished processing BAM request
2018-11-19 22:29:23 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:29:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562945   8 58441 80963029 A  ERROR'
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:29:23 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:29:23 [DEBUG   ] spawn_process: at 2018-11-19 22:29:23 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:29:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 22:29:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:29:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:29:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:29:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:29:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:29:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:29:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:29:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:29:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:29:28 [DEBUG   ] Finished updating board flags in 1.446 seconds
2018-11-19 22:29:29 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:29:49 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:30:00 [DEBUG   ] Got command BAM from MCS: ref #562950
2018-11-19 22:30:00 [INFO    ] Finished processing BAM request
2018-11-19 22:30:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:30:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562950   8 58441 81000029 A  ERROR'
2018-11-19 22:30:00 [DEBUG   ] Got command BAM from MCS: ref #562949
2018-11-19 22:30:00 [INFO    ] Finished processing BAM request
2018-11-19 22:30:00 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:30:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562949   8 58441 81000033 A  ERROR'
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:30:00 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:30:00 [DEBUG   ] spawn_process: at 2018-11-19 22:30:00 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:30:01 [DEBUG   ] Got command RPT from MCS: ref #562951
2018-11-19 22:30:01 [DEBUG   ] summary = ERROR
2018-11-19 22:30:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562951  13 58441 81001737 A  ERRORERROR'
2018-11-19 22:30:01 [DEBUG   ] Got command RPT from MCS: ref #562952
2018-11-19 22:30:01 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:30:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562952  77 58441 81001941 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:30:02 [DEBUG   ] Got command RPT from MCS: ref #562953
2018-11-19 22:30:02 [DEBUG   ] summary = ERROR
2018-11-19 22:30:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562953  13 58441 81002149 A  ERRORERROR'
2018-11-19 22:30:02 [DEBUG   ] Got command RPT from MCS: ref #562955
2018-11-19 22:30:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:30:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562955  77 58441 81002341 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:30:02 [DEBUG   ] Got command RPT from MCS: ref #562956
2018-11-19 22:30:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:30:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562956  12 58441 81002545 A  ERRORB£ff'
2018-11-19 22:30:09 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 22:30:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:30:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:30:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:30:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:30:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:30:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:30:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:30:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:30:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:30:28 [DEBUG   ] Finished updating board flags in 1.414 seconds
2018-11-19 22:30:29 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 22:30:37 [DEBUG   ] Got command BAM from MCS: ref #562961
2018-11-19 22:30:37 [INFO    ] Finished processing BAM request
2018-11-19 22:30:37 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:30:37 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562961   8 58441 81037025 A  ERROR'
2018-11-19 22:30:37 [DEBUG   ] Got command BAM from MCS: ref #562962
2018-11-19 22:30:37 [INFO    ] Finished processing BAM request
2018-11-19 22:30:37 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:30:37 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562962   8 58441 81037029 A  ERROR'
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:30:37 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:30:37 [DEBUG   ] spawn_process: at 2018-11-19 22:30:37 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:30:46 [DEBUG   ] Got command RPT from MCS: ref #562963
2018-11-19 22:30:46 [DEBUG   ] summary = ERROR
2018-11-19 22:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562963  13 58441 81046525 A  ERRORERROR'
2018-11-19 22:30:46 [DEBUG   ] Got command RPT from MCS: ref #562964
2018-11-19 22:30:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   562964  77 58441 81046526 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:30:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:31:09 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:31:14 [DEBUG   ] Got command BAM from MCS: ref #562985
2018-11-19 22:31:14 [INFO    ] Finished processing BAM request
2018-11-19 22:31:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:31:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562985   8 58441 81074029 A  ERROR'
2018-11-19 22:31:14 [DEBUG   ] Got command BAM from MCS: ref #562984
2018-11-19 22:31:14 [INFO    ] Finished processing BAM request
2018-11-19 22:31:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:31:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562984   8 58441 81074033 A  ERROR'
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:31:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:31:14 [DEBUG   ] spawn_process: at 2018-11-19 22:31:14 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:31:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:31:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:31:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:31:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:31:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:31:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:31:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:31:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:31:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:31:28 [DEBUG   ] Finished updating board flags in 1.338 seconds
2018-11-19 22:31:29 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:31:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:31:51 [DEBUG   ] Got command BAM from MCS: ref #562987
2018-11-19 22:31:51 [INFO    ] Finished processing BAM request
2018-11-19 22:31:51 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:31:51 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562987   8 58441 81111033 A  ERROR'
2018-11-19 22:31:51 [DEBUG   ] Got command BAM from MCS: ref #562986
2018-11-19 22:31:51 [INFO    ] Finished processing BAM request
2018-11-19 22:31:51 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:31:51 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562986   8 58441 81111037 A  ERROR'
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:31:51 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:31:51 [DEBUG   ] spawn_process: at 2018-11-19 22:31:51 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:32:09 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:32:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:32:28 [DEBUG   ] Got command BAM from MCS: ref #562988
2018-11-19 22:32:28 [INFO    ] Finished processing BAM request
2018-11-19 22:32:28 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:32:28 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562988   8 58441 81148037 A  ERROR'
2018-11-19 22:32:28 [DEBUG   ] Got command BAM from MCS: ref #562989
2018-11-19 22:32:28 [INFO    ] Finished processing BAM request
2018-11-19 22:32:28 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:32:28 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562989   8 58441 81148045 A  ERROR'
2018-11-19 22:32:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:32:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:32:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:32:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:32:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:32:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:32:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:32:28 [DEBUG   ] Finished updating board flags in 1.210 seconds
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:32:28 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:32:28 [DEBUG   ] spawn_process: at 2018-11-19 22:32:28 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:32:29 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 22:32:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:33:05 [DEBUG   ] Got command BAM from MCS: ref #562991
2018-11-19 22:33:05 [INFO    ] Finished processing BAM request
2018-11-19 22:33:05 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:33:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562991   8 58441 81185025 A  ERROR'
2018-11-19 22:33:05 [DEBUG   ] Got command BAM from MCS: ref #562990
2018-11-19 22:33:05 [INFO    ] Finished processing BAM request
2018-11-19 22:33:05 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:33:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562990   8 58441 81185029 A  ERROR'
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:33:05 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:33:05 [DEBUG   ] spawn_process: at 2018-11-19 22:33:05 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:33:09 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:33:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:33:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:33:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 22:33:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:33:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:33:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:33:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:33:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:33:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:33:28 [DEBUG   ] Finished updating board flags in 1.198 seconds
2018-11-19 22:33:29 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:33:42 [DEBUG   ] Got command BAM from MCS: ref #562993
2018-11-19 22:33:42 [INFO    ] Finished processing BAM request
2018-11-19 22:33:42 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:33:42 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562993   8 58441 81222025 A  ERROR'
2018-11-19 22:33:42 [DEBUG   ] Got command BAM from MCS: ref #562992
2018-11-19 22:33:42 [INFO    ] Finished processing BAM request
2018-11-19 22:33:42 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:33:42 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562992   8 58441 81222029 A  ERROR'
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:33:42 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:33:42 [DEBUG   ] spawn_process: at 2018-11-19 22:33:42 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:33:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:34:09 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:34:19 [DEBUG   ] Got command BAM from MCS: ref #562998
2018-11-19 22:34:19 [INFO    ] Finished processing BAM request
2018-11-19 22:34:19 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:34:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562998   8 58441 81259029 A  ERROR'
2018-11-19 22:34:19 [DEBUG   ] Got command BAM from MCS: ref #562999
2018-11-19 22:34:19 [INFO    ] Finished processing BAM request
2018-11-19 22:34:19 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:34:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   562999   8 58441 81259033 A  ERROR'
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:34:19 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:34:19 [DEBUG   ] spawn_process: at 2018-11-19 22:34:19 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:34:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:34:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:34:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 22:34:29 [DEBUG   ] Finished updating ant stats in 1.030 seconds
2018-11-19 22:34:29 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:34:29 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:34:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:34:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:34:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:34:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:34:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:34:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:34:29 [DEBUG   ] Finished updating board flags in 2.027 seconds
2018-11-19 22:34:49 [DEBUG   ] Finished updating ant stats in 0.974 seconds
2018-11-19 22:34:56 [DEBUG   ] Got command BAM from MCS: ref #563000
2018-11-19 22:34:56 [INFO    ] Finished processing BAM request
2018-11-19 22:34:56 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:34:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563000   8 58441 81296029 A  ERROR'
2018-11-19 22:34:56 [DEBUG   ] Got command BAM from MCS: ref #563001
2018-11-19 22:34:56 [INFO    ] Finished processing BAM request
2018-11-19 22:34:56 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:34:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563001   8 58441 81296033 A  ERROR'
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:34:56 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:34:56 [DEBUG   ] spawn_process: at 2018-11-19 22:34:56 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:35:01 [DEBUG   ] Got command RPT from MCS: ref #563002
2018-11-19 22:35:01 [DEBUG   ] summary = ERROR
2018-11-19 22:35:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563002  13 58441 81301829 A  ERRORERROR'
2018-11-19 22:35:02 [DEBUG   ] Got command RPT from MCS: ref #563003
2018-11-19 22:35:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:35:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563003  77 58441 81302033 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:35:02 [DEBUG   ] Got command RPT from MCS: ref #563004
2018-11-19 22:35:02 [DEBUG   ] summary = ERROR
2018-11-19 22:35:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563004  13 58441 81302225 A  ERRORERROR'
2018-11-19 22:35:02 [DEBUG   ] Got command RPT from MCS: ref #563005
2018-11-19 22:35:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:35:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563005  77 58441 81302429 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:35:02 [DEBUG   ] Got command RPT from MCS: ref #563006
2018-11-19 22:35:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:35:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563006  12 58441 81302633 A  ERRORB£ff'
2018-11-19 22:35:09 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:35:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:35:28 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:35:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:35:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:35:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:35:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:35:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:35:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:35:28 [DEBUG   ] Finished updating board flags in 1.427 seconds
2018-11-19 22:35:29 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 22:35:33 [DEBUG   ] Got command BAM from MCS: ref #563007
2018-11-19 22:35:33 [INFO    ] Finished processing BAM request
2018-11-19 22:35:33 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:35:33 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563007   8 58441 81333029 A  ERROR'
2018-11-19 22:35:33 [DEBUG   ] Got command BAM from MCS: ref #563008
2018-11-19 22:35:33 [INFO    ] Finished processing BAM request
2018-11-19 22:35:33 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:35:33 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563008   8 58441 81333033 A  ERROR'
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:35:33 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:35:33 [DEBUG   ] spawn_process: at 2018-11-19 22:35:33 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:35:46 [DEBUG   ] Got command RPT from MCS: ref #563009
2018-11-19 22:35:46 [DEBUG   ] summary = ERROR
2018-11-19 22:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563009  13 58441 81346525 A  ERRORERROR'
2018-11-19 22:35:46 [DEBUG   ] Got command RPT from MCS: ref #563010
2018-11-19 22:35:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563010  77 58441 81346526 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:35:49 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:36:09 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:36:10 [DEBUG   ] Got command BAM from MCS: ref #563027
2018-11-19 22:36:10 [INFO    ] Finished processing BAM request
2018-11-19 22:36:10 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:36:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563027   8 58441 81370029 A  ERROR'
2018-11-19 22:36:10 [DEBUG   ] Got command BAM from MCS: ref #563026
2018-11-19 22:36:10 [INFO    ] Finished processing BAM request
2018-11-19 22:36:10 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:36:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563026   8 58441 81370033 A  ERROR'
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:36:10 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:36:10 [DEBUG   ] spawn_process: at 2018-11-19 22:36:10 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:36:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:36:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:36:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:36:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:36:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:36:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:36:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:36:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:36:28 [DEBUG   ] Finished updating board flags in 1.415 seconds
2018-11-19 22:36:29 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 22:36:47 [DEBUG   ] Got command BAM from MCS: ref #563029
2018-11-19 22:36:47 [INFO    ] Finished processing BAM request
2018-11-19 22:36:47 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:36:47 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563029   8 58441 81407029 A  ERROR'
2018-11-19 22:36:47 [DEBUG   ] Got command BAM from MCS: ref #563028
2018-11-19 22:36:47 [INFO    ] Finished processing BAM request
2018-11-19 22:36:47 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:36:47 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563028   8 58441 81407033 A  ERROR'
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:36:47 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:36:47 [DEBUG   ] spawn_process: at 2018-11-19 22:36:47 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:36:49 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:37:09 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:37:24 [DEBUG   ] Got command BAM from MCS: ref #563034
2018-11-19 22:37:24 [INFO    ] Finished processing BAM request
2018-11-19 22:37:24 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:37:24 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563034   8 58441 81444033 A  ERROR'
2018-11-19 22:37:24 [DEBUG   ] Got command BAM from MCS: ref #563035
2018-11-19 22:37:24 [INFO    ] Finished processing BAM request
2018-11-19 22:37:24 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:37:24 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563035   8 58441 81444037 A  ERROR'
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:37:24 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:37:24 [DEBUG   ] spawn_process: at 2018-11-19 22:37:24 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:37:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:37:28 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:37:28 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:37:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:37:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:37:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:37:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:37:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:37:28 [DEBUG   ] Finished updating board flags in 1.373 seconds
2018-11-19 22:37:29 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:37:49 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:38:01 [DEBUG   ] Got command BAM from MCS: ref #563037
2018-11-19 22:38:01 [INFO    ] Finished processing BAM request
2018-11-19 22:38:01 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:38:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563037   8 58441 81481025 A  ERROR'
2018-11-19 22:38:01 [DEBUG   ] Got command BAM from MCS: ref #563036
2018-11-19 22:38:01 [INFO    ] Finished processing BAM request
2018-11-19 22:38:01 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:38:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563036   8 58441 81481029 A  ERROR'
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:38:01 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:38:01 [DEBUG   ] spawn_process: at 2018-11-19 22:38:01 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:38:09 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 22:38:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:38:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:38:29 [DEBUG   ] Finished updating ant stats in 0.630 seconds
2018-11-19 22:38:29 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:38:29 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:38:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:38:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:38:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:38:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:38:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:38:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:38:29 [DEBUG   ] Finished updating board flags in 1.796 seconds
2018-11-19 22:38:38 [DEBUG   ] Got command BAM from MCS: ref #563038
2018-11-19 22:38:38 [INFO    ] Finished processing BAM request
2018-11-19 22:38:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:38:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563038   8 58441 81518025 A  ERROR'
2018-11-19 22:38:38 [DEBUG   ] Got command BAM from MCS: ref #563039
2018-11-19 22:38:38 [INFO    ] Finished processing BAM request
2018-11-19 22:38:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:38:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563039   8 58441 81518029 A  ERROR'
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:38:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:38:38 [DEBUG   ] spawn_process: at 2018-11-19 22:38:38 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:38:49 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-19 22:39:10 [DEBUG   ] Finished updating ant stats in 1.402 seconds
2018-11-19 22:39:14 [DEBUG   ] Got command BAM from MCS: ref #563040
2018-11-19 22:39:14 [INFO    ] Finished processing BAM request
2018-11-19 22:39:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:39:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563040   8 58441 81554033 A  ERROR'
2018-11-19 22:39:14 [DEBUG   ] Got command BAM from MCS: ref #563041
2018-11-19 22:39:14 [INFO    ] Finished processing BAM request
2018-11-19 22:39:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:39:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563041   8 58441 81554037 A  ERROR'
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:39:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:39:14 [DEBUG   ] spawn_process: at 2018-11-19 22:39:14 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:39:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:39:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:39:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:39:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:39:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:39:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:39:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:39:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:39:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:39:28 [DEBUG   ] Finished updating board flags in 1.018 seconds
2018-11-19 22:39:29 [DEBUG   ] Finished updating ant stats in 0.398 seconds
2018-11-19 22:39:50 [DEBUG   ] Got command BAM from MCS: ref #563044
2018-11-19 22:39:50 [INFO    ] Finished processing BAM request
2018-11-19 22:39:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:39:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563044   8 58441 81590033 A  ERROR'
2018-11-19 22:39:50 [DEBUG   ] Got command BAM from MCS: ref #563045
2018-11-19 22:39:50 [INFO    ] Finished processing BAM request
2018-11-19 22:39:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:39:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563045   8 58441 81590037 A  ERROR'
2018-11-19 22:39:50 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:39:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:39:50 [DEBUG   ] spawn_process: at 2018-11-19 22:39:50 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:40:01 [DEBUG   ] Got command RPT from MCS: ref #563046
2018-11-19 22:40:01 [DEBUG   ] summary = ERROR
2018-11-19 22:40:01 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563046  13 58441 81601921 A  ERRORERROR'
2018-11-19 22:40:02 [DEBUG   ] Got command RPT from MCS: ref #563047
2018-11-19 22:40:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:40:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563047  77 58441 81602113 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:40:02 [DEBUG   ] Got command RPT from MCS: ref #563048
2018-11-19 22:40:02 [DEBUG   ] summary = ERROR
2018-11-19 22:40:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563048  13 58441 81602317 A  ERRORERROR'
2018-11-19 22:40:02 [DEBUG   ] Got command RPT from MCS: ref #563050
2018-11-19 22:40:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:40:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563050  77 58441 81602521 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:40:02 [DEBUG   ] Got command RPT from MCS: ref #563051
2018-11-19 22:40:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:40:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563051  12 58441 81602713 A  ERRORB£ff'
2018-11-19 22:40:10 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 22:40:26 [DEBUG   ] Got command BAM from MCS: ref #563061
2018-11-19 22:40:26 [INFO    ] Finished processing BAM request
2018-11-19 22:40:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:40:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563061   8 58441 81626033 A  ERROR'
2018-11-19 22:40:26 [DEBUG   ] Got command BAM from MCS: ref #563060
2018-11-19 22:40:26 [INFO    ] Finished processing BAM request
2018-11-19 22:40:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:40:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563060   8 58441 81626037 A  ERROR'
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:40:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:40:26 [DEBUG   ] spawn_process: at 2018-11-19 22:40:26 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:40:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:40:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:40:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:40:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:40:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:40:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:40:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:40:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:40:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:40:28 [DEBUG   ] Finished updating board flags in 1.003 seconds
2018-11-19 22:40:30 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 22:40:46 [DEBUG   ] Got command RPT from MCS: ref #563062
2018-11-19 22:40:46 [DEBUG   ] summary = ERROR
2018-11-19 22:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563062  13 58441 81646529 A  ERRORERROR'
2018-11-19 22:40:46 [DEBUG   ] Got command RPT from MCS: ref #563063
2018-11-19 22:40:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563063  77 58441 81646530 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:40:50 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:41:02 [DEBUG   ] Got command BAM from MCS: ref #563080
2018-11-19 22:41:02 [INFO    ] Finished processing BAM request
2018-11-19 22:41:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:41:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563080   8 58441 81662025 A  ERROR'
2018-11-19 22:41:02 [DEBUG   ] Got command BAM from MCS: ref #563079
2018-11-19 22:41:02 [INFO    ] Finished processing BAM request
2018-11-19 22:41:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:41:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563079   8 58441 81662029 A  ERROR'
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:41:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:41:02 [DEBUG   ] spawn_process: at 2018-11-19 22:41:02 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:41:10 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:41:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:41:28 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:41:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:41:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:41:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:41:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:41:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:41:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:41:28 [DEBUG   ] Finished updating board flags in 1.051 seconds
2018-11-19 22:41:30 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 22:41:38 [DEBUG   ] Got command BAM from MCS: ref #563082
2018-11-19 22:41:38 [INFO    ] Finished processing BAM request
2018-11-19 22:41:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:41:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563082   8 58441 81698029 A  ERROR'
2018-11-19 22:41:38 [DEBUG   ] Got command BAM from MCS: ref #563081
2018-11-19 22:41:38 [INFO    ] Finished processing BAM request
2018-11-19 22:41:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:41:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563081   8 58441 81698033 A  ERROR'
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:41:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:41:38 [DEBUG   ] spawn_process: at 2018-11-19 22:41:38 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:41:50 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:42:10 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:42:14 [DEBUG   ] Got command BAM from MCS: ref #563083
2018-11-19 22:42:14 [INFO    ] Finished processing BAM request
2018-11-19 22:42:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:42:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563083   8 58441 81734025 A  ERROR'
2018-11-19 22:42:14 [DEBUG   ] Got command BAM from MCS: ref #563084
2018-11-19 22:42:14 [INFO    ] Finished processing BAM request
2018-11-19 22:42:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:42:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563084   8 58441 81734029 A  ERROR'
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:42:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:42:14 [DEBUG   ] spawn_process: at 2018-11-19 22:42:14 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:42:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:42:28 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:42:28 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:42:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:42:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:42:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:42:28 [WARNING ] (13, 1, 'bot')
2018-11-19 22:42:28 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:42:28 [DEBUG   ] Finished updating board flags in 0.918 seconds
2018-11-19 22:42:30 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 22:42:50 [DEBUG   ] Got command BAM from MCS: ref #563085
2018-11-19 22:42:50 [INFO    ] Finished processing BAM request
2018-11-19 22:42:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:42:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563085   8 58441 81770025 A  ERROR'
2018-11-19 22:42:50 [DEBUG   ] Got command BAM from MCS: ref #563086
2018-11-19 22:42:50 [INFO    ] Finished processing BAM request
2018-11-19 22:42:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:42:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563086   8 58441 81770029 A  ERROR'
2018-11-19 22:42:50 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:42:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:42:50 [DEBUG   ] spawn_process: at 2018-11-19 22:42:50 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:43:10 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:43:26 [DEBUG   ] Got command BAM from MCS: ref #563092
2018-11-19 22:43:26 [INFO    ] Finished processing BAM request
2018-11-19 22:43:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:43:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563092   8 58441 81806025 A  ERROR'
2018-11-19 22:43:26 [DEBUG   ] Got command BAM from MCS: ref #563091
2018-11-19 22:43:26 [INFO    ] Finished processing BAM request
2018-11-19 22:43:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:43:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563091   8 58441 81806029 A  ERROR'
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:43:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:43:26 [DEBUG   ] spawn_process: at 2018-11-19 22:43:26 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:43:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:43:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:43:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 22:43:29 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:43:29 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:43:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:43:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:43:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:43:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:43:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:43:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:43:29 [DEBUG   ] Finished updating board flags in 1.385 seconds
2018-11-19 22:43:30 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:43:50 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 22:44:02 [DEBUG   ] Got command BAM from MCS: ref #563094
2018-11-19 22:44:02 [INFO    ] Finished processing BAM request
2018-11-19 22:44:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:44:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563094   8 58441 81842025 A  ERROR'
2018-11-19 22:44:02 [DEBUG   ] Got command BAM from MCS: ref #563093
2018-11-19 22:44:02 [INFO    ] Finished processing BAM request
2018-11-19 22:44:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:44:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563093   8 58441 81842029 A  ERROR'
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:44:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:44:02 [DEBUG   ] spawn_process: at 2018-11-19 22:44:02 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:44:10 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:44:28 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 22:44:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:44:29 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:44:29 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:44:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:44:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:44:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:44:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:44:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:44:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:44:29 [DEBUG   ] Finished updating board flags in 1.227 seconds
2018-11-19 22:44:30 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:44:38 [DEBUG   ] Got command BAM from MCS: ref #563095
2018-11-19 22:44:38 [INFO    ] Finished processing BAM request
2018-11-19 22:44:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:44:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563095   8 58441 81878029 A  ERROR'
2018-11-19 22:44:38 [DEBUG   ] Got command BAM from MCS: ref #563096
2018-11-19 22:44:38 [INFO    ] Finished processing BAM request
2018-11-19 22:44:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:44:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563096   8 58441 81878033 A  ERROR'
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:44:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:44:38 [DEBUG   ] spawn_process: at 2018-11-19 22:44:38 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:44:50 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:45:02 [DEBUG   ] Got command RPT from MCS: ref #563097
2018-11-19 22:45:02 [DEBUG   ] summary = ERROR
2018-11-19 22:45:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563097  13 58441 81902005 A  ERRORERROR'
2018-11-19 22:45:02 [DEBUG   ] Got command RPT from MCS: ref #563098
2018-11-19 22:45:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:45:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563098  77 58441 81902197 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:45:02 [DEBUG   ] Got command RPT from MCS: ref #563099
2018-11-19 22:45:02 [DEBUG   ] summary = ERROR
2018-11-19 22:45:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563099  13 58441 81902401 A  ERRORERROR'
2018-11-19 22:45:02 [DEBUG   ] Got command RPT from MCS: ref #563100
2018-11-19 22:45:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:45:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563100  77 58441 81902605 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:45:02 [DEBUG   ] Got command RPT from MCS: ref #563101
2018-11-19 22:45:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:45:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563101  12 58441 81902809 A  ERRORB¤ff'
2018-11-19 22:45:10 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:45:14 [DEBUG   ] Got command BAM from MCS: ref #563102
2018-11-19 22:45:14 [INFO    ] Finished processing BAM request
2018-11-19 22:45:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:45:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563102   8 58441 81914033 A  ERROR'
2018-11-19 22:45:14 [DEBUG   ] Got command BAM from MCS: ref #563103
2018-11-19 22:45:14 [INFO    ] Finished processing BAM request
2018-11-19 22:45:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:45:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563103   8 58441 81914037 A  ERROR'
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:45:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:45:14 [DEBUG   ] spawn_process: at 2018-11-19 22:45:14 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:45:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 22:45:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:45:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:45:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:45:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 22:45:29 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:45:29 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:45:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:45:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:45:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:45:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:45:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:45:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:45:29 [DEBUG   ] Finished updating board flags in 1.154 seconds
2018-11-19 22:45:30 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 22:45:46 [DEBUG   ] Got command RPT from MCS: ref #563104
2018-11-19 22:45:46 [DEBUG   ] summary = ERROR
2018-11-19 22:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563104  13 58441 81946529 A  ERRORERROR'
2018-11-19 22:45:46 [DEBUG   ] Got command RPT from MCS: ref #563105
2018-11-19 22:45:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563105  77 58441 81946530 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:45:50 [DEBUG   ] Got command BAM from MCS: ref #563121
2018-11-19 22:45:50 [INFO    ] Finished processing BAM request
2018-11-19 22:45:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:45:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563121   8 58441 81950025 A  ERROR'
2018-11-19 22:45:50 [DEBUG   ] Got command BAM from MCS: ref #563122
2018-11-19 22:45:50 [INFO    ] Finished processing BAM request
2018-11-19 22:45:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:45:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563122   8 58441 81950029 A  ERROR'
2018-11-19 22:45:50 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:45:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:45:50 [DEBUG   ] spawn_process: at 2018-11-19 22:45:50 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:46:10 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:46:26 [DEBUG   ] Got command BAM from MCS: ref #563128
2018-11-19 22:46:26 [INFO    ] Finished processing BAM request
2018-11-19 22:46:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:46:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563128   8 58441 81986029 A  ERROR'
2018-11-19 22:46:26 [DEBUG   ] Got command BAM from MCS: ref #563127
2018-11-19 22:46:26 [INFO    ] Finished processing BAM request
2018-11-19 22:46:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:46:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563127   8 58441 81986033 A  ERROR'
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:46:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:46:26 [DEBUG   ] spawn_process: at 2018-11-19 22:46:26 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:46:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:46:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 22:46:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 22:46:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:46:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 22:46:29 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:46:29 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:46:29 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:46:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:46:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:46:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:46:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:46:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:46:29 [DEBUG   ] Finished updating board flags in 0.995 seconds
2018-11-19 22:46:30 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:46:50 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 22:47:02 [DEBUG   ] Got command BAM from MCS: ref #563129
2018-11-19 22:47:02 [INFO    ] Finished processing BAM request
2018-11-19 22:47:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:47:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563129   8 58441 82022026 A  ERROR'
2018-11-19 22:47:02 [DEBUG   ] Got command BAM from MCS: ref #563130
2018-11-19 22:47:02 [INFO    ] Finished processing BAM request
2018-11-19 22:47:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:47:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563130   8 58441 82022030 A  ERROR'
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:47:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:47:02 [DEBUG   ] spawn_process: at 2018-11-19 22:47:02 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:47:10 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:47:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:47:29 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:47:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:47:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:47:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:47:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:47:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:47:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:47:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:47:29 [DEBUG   ] Finished updating board flags in 1.301 seconds
2018-11-19 22:47:30 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 22:47:38 [DEBUG   ] Got command BAM from MCS: ref #563131
2018-11-19 22:47:38 [INFO    ] Finished processing BAM request
2018-11-19 22:47:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:47:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563131   8 58441 82058034 A  ERROR'
2018-11-19 22:47:38 [DEBUG   ] Got command BAM from MCS: ref #563132
2018-11-19 22:47:38 [INFO    ] Finished processing BAM request
2018-11-19 22:47:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:47:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563132   8 58441 82058038 A  ERROR'
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:47:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:47:38 [DEBUG   ] spawn_process: at 2018-11-19 22:47:38 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:47:50 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:48:10 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:48:14 [DEBUG   ] Got command BAM from MCS: ref #563133
2018-11-19 22:48:14 [INFO    ] Finished processing BAM request
2018-11-19 22:48:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:48:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563133   8 58441 82094030 A  ERROR'
2018-11-19 22:48:14 [DEBUG   ] Got command BAM from MCS: ref #563134
2018-11-19 22:48:14 [INFO    ] Finished processing BAM request
2018-11-19 22:48:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:48:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563134   8 58441 82094034 A  ERROR'
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:48:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:48:14 [DEBUG   ] spawn_process: at 2018-11-19 22:48:14 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:48:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 22:48:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:48:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 22:48:30 [DEBUG   ] Finished updating ant stats in 0.630 seconds
2018-11-19 22:48:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 22:48:30 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:48:30 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:48:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:48:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:48:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:48:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:48:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:48:30 [DEBUG   ] Finished updating board flags in 1.766 seconds
2018-11-19 22:48:50 [DEBUG   ] Got command BAM from MCS: ref #563135
2018-11-19 22:48:50 [INFO    ] Finished processing BAM request
2018-11-19 22:48:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:48:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563135   8 58441 82130026 A  ERROR'
2018-11-19 22:48:50 [DEBUG   ] Got command BAM from MCS: ref #563136
2018-11-19 22:48:50 [INFO    ] Finished processing BAM request
2018-11-19 22:48:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:48:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563136   8 58441 82130030 A  ERROR'
2018-11-19 22:48:50 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1'] to hostname lwa27
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp5 1']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2'] to hostname lwa24
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp5 2']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3'] to hostname lwa23
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp5 3']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4'] to hostname lwa18
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp5 4']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5'] to hostname lwa10
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp5 5']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6'] to hostname lwa32
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp5 6']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7'] to hostname lwa11
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp5 7']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8'] to hostname lwa30
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp5 8']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9'] to hostname lwa12
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp5 9']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10'] to hostname lwa8
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp5 10']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11'] to hostname lwa13
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp5 11']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12'] to hostname lwa26
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp5 12']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13'] to hostname lwa14
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp5 13']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14'] to hostname lwa22
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp5 14']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15'] to hostname lwa20
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp5 15']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16'] to hostname lwa1
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp5 16']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17'] to hostname lwa5
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp5 17']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18'] to hostname lwa25
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp5 18']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19'] to hostname lwa35
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp5 19']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20'] to hostname lwa19
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp5 20']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21'] to hostname lwa21
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp5 21']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22'] to hostname lwa6
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp5 22']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23'] to hostname lwa7
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp5 23']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24'] to hostname lwa33
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp5 24']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25'] to hostname lwa15
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp5 25']
2018-11-19 22:48:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26'] to hostname lwa28
2018-11-19 22:48:50 [DEBUG   ] spawn_process: at 2018-11-19 22:48:50 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp5 26']
2018-11-19 22:49:10 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:49:26 [DEBUG   ] Got command BAM from MCS: ref #563142
2018-11-19 22:49:26 [INFO    ] Finished processing BAM request
2018-11-19 22:49:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:49:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563142   8 58441 82166033 A  ERROR'
2018-11-19 22:49:26 [DEBUG   ] Got command BAM from MCS: ref #563141
2018-11-19 22:49:26 [INFO    ] Finished processing BAM request
2018-11-19 22:49:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:49:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563141   8 58441 82166037 A  ERROR'
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:49:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:49:26 [DEBUG   ] spawn_process: at 2018-11-19 22:49:26 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:49:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 22:49:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:49:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:49:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-19 22:49:29 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:49:29 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:49:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:49:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:49:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:49:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:49:29 [WARNING ] (13, 1, 'bot')
2018-11-19 22:49:29 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:49:29 [DEBUG   ] Finished updating board flags in 0.991 seconds
2018-11-19 22:49:30 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:49:50 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:50:02 [DEBUG   ] Got command BAM from MCS: ref #563146
2018-11-19 22:50:02 [INFO    ] Finished processing BAM request
2018-11-19 22:50:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:50:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563146   8 58441 82202025 A  ERROR'
2018-11-19 22:50:02 [DEBUG   ] Got command BAM from MCS: ref #563145
2018-11-19 22:50:02 [INFO    ] Finished processing BAM request
2018-11-19 22:50:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:50:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563145   8 58441 82202029 A  ERROR'
2018-11-19 22:50:02 [DEBUG   ] Got command RPT from MCS: ref #563147
2018-11-19 22:50:02 [DEBUG   ] summary = ERROR
2018-11-19 22:50:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563147  13 58441 82202089 A  ERRORERROR'
2018-11-19 22:50:02 [DEBUG   ] Got command RPT from MCS: ref #563148
2018-11-19 22:50:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:50:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563148  77 58441 82202281 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:50:02 [DEBUG   ] Got command RPT from MCS: ref #563149
2018-11-19 22:50:02 [DEBUG   ] summary = ERROR
2018-11-19 22:50:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563149  13 58441 82202485 A  ERRORERROR'
2018-11-19 22:50:02 [DEBUG   ] Got command RPT from MCS: ref #563151
2018-11-19 22:50:02 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:50:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563151  77 58441 82202689 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:50:02 [DEBUG   ] Got command RPT from MCS: ref #563152
2018-11-19 22:50:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:50:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563152  12 58441 82202894 A  ERRORB£ff'
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1'] to hostname lwa27
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp3 1']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2'] to hostname lwa24
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp3 2']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3'] to hostname lwa23
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp3 3']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4'] to hostname lwa18
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp3 4']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5'] to hostname lwa10
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp3 5']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6'] to hostname lwa32
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp3 6']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7'] to hostname lwa11
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp3 7']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8'] to hostname lwa30
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp3 8']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9'] to hostname lwa12
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp3 9']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10'] to hostname lwa8
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp3 10']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11'] to hostname lwa13
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp3 11']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12'] to hostname lwa26
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp3 12']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13'] to hostname lwa14
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp3 13']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14'] to hostname lwa22
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp3 14']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15'] to hostname lwa20
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp3 15']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16'] to hostname lwa1
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp3 16']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17'] to hostname lwa5
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp3 17']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18'] to hostname lwa25
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp3 18']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19'] to hostname lwa35
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp3 19']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20'] to hostname lwa19
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp3 20']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21'] to hostname lwa21
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp3 21']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22'] to hostname lwa6
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp3 22']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23'] to hostname lwa7
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp3 23']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24'] to hostname lwa33
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp3 24']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25'] to hostname lwa15
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp3 25']
2018-11-19 22:50:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26'] to hostname lwa28
2018-11-19 22:50:02 [DEBUG   ] spawn_process: at 2018-11-19 22:50:02 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp3 26']
2018-11-19 22:50:10 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:50:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 22:50:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:50:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 22:50:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:50:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 22:50:30 [DEBUG   ] Finished updating ant stats in 0.430 seconds
2018-11-19 22:50:30 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:50:30 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa7, chip 2
2018-11-19 22:50:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:50:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:50:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:50:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:50:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:50:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:50:30 [DEBUG   ] Finished updating board flags in 1.416 seconds
2018-11-19 22:50:38 [DEBUG   ] Got command BAM from MCS: ref #563157
2018-11-19 22:50:38 [INFO    ] Finished processing BAM request
2018-11-19 22:50:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:50:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563157   8 58441 82238029 A  ERROR'
2018-11-19 22:50:38 [DEBUG   ] Got command BAM from MCS: ref #563158
2018-11-19 22:50:38 [INFO    ] Finished processing BAM request
2018-11-19 22:50:38 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:50:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563158   8 58441 82238033 A  ERROR'
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:50:38 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:50:38 [DEBUG   ] spawn_process: at 2018-11-19 22:50:38 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:50:46 [DEBUG   ] Got command RPT from MCS: ref #563159
2018-11-19 22:50:46 [DEBUG   ] summary = ERROR
2018-11-19 22:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563159  13 58441 82246521 A  ERRORERROR'
2018-11-19 22:50:46 [DEBUG   ] Got command RPT from MCS: ref #563160
2018-11-19 22:50:46 [DEBUG   ] info = T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563160  77 58441 82246522 A  ERRORT_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:50:50 [DEBUG   ] Finished updating ant stats in 0.374 seconds
2018-11-19 22:51:11 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 22:51:14 [DEBUG   ] Got command BAM from MCS: ref #563177
2018-11-19 22:51:14 [INFO    ] Finished processing BAM request
2018-11-19 22:51:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:51:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563177   8 58441 82274029 A  ERROR'
2018-11-19 22:51:14 [DEBUG   ] Got command BAM from MCS: ref #563176
2018-11-19 22:51:14 [INFO    ] Finished processing BAM request
2018-11-19 22:51:14 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:51:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563176   8 58441 82274033 A  ERROR'
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1'] to hostname lwa27
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp2 1']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2'] to hostname lwa24
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp2 2']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3'] to hostname lwa23
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp2 3']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4'] to hostname lwa18
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp2 4']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5'] to hostname lwa10
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp2 5']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6'] to hostname lwa32
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp2 6']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7'] to hostname lwa11
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp2 7']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8'] to hostname lwa30
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp2 8']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9'] to hostname lwa12
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp2 9']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10'] to hostname lwa8
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp2 10']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11'] to hostname lwa13
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp2 11']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12'] to hostname lwa26
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp2 12']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13'] to hostname lwa14
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp2 13']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14'] to hostname lwa22
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp2 14']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15'] to hostname lwa20
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp2 15']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16'] to hostname lwa1
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp2 16']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17'] to hostname lwa5
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp2 17']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18'] to hostname lwa25
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp2 18']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19'] to hostname lwa35
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp2 19']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20'] to hostname lwa19
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp2 20']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21'] to hostname lwa21
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp2 21']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22'] to hostname lwa6
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp2 22']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23'] to hostname lwa7
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp2 23']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24'] to hostname lwa33
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp2 24']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25'] to hostname lwa15
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp2 25']
2018-11-19 22:51:14 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26'] to hostname lwa28
2018-11-19 22:51:14 [DEBUG   ] spawn_process: at 2018-11-19 22:51:14 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp2 26']
2018-11-19 22:51:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 22:51:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:51:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:51:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 22:51:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 22:51:30 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:51:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:51:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:51:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:51:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:51:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:51:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:51:30 [DEBUG   ] Finished updating board flags in 1.376 seconds
2018-11-19 22:51:31 [DEBUG   ] Finished updating ant stats in 1.174 seconds
2018-11-19 22:51:50 [DEBUG   ] Got command BAM from MCS: ref #563178
2018-11-19 22:51:50 [INFO    ] Finished processing BAM request
2018-11-19 22:51:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:51:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563178   8 58441 82310033 A  ERROR'
2018-11-19 22:51:50 [DEBUG   ] Got command BAM from MCS: ref #563179
2018-11-19 22:51:50 [INFO    ] Finished processing BAM request
2018-11-19 22:51:50 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:51:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563179   8 58441 82310037 A  ERROR'
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:51:50 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:51:50 [DEBUG   ] spawn_process: at 2018-11-19 22:51:50 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:51:51 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 22:52:11 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:52:26 [DEBUG   ] Got command BAM from MCS: ref #563185
2018-11-19 22:52:26 [INFO    ] Finished processing BAM request
2018-11-19 22:52:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:52:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563185   8 58441 82346025 A  ERROR'
2018-11-19 22:52:26 [DEBUG   ] Got command BAM from MCS: ref #563184
2018-11-19 22:52:26 [INFO    ] Finished processing BAM request
2018-11-19 22:52:26 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:52:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563184   8 58441 82346029 A  ERROR'
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1'] to hostname lwa27
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp1 1']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2'] to hostname lwa24
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp1 2']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3'] to hostname lwa23
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp1 3']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4'] to hostname lwa18
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp1 4']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5'] to hostname lwa10
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp1 5']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6'] to hostname lwa32
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp1 6']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7'] to hostname lwa11
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp1 7']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8'] to hostname lwa30
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp1 8']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9'] to hostname lwa12
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp1 9']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10'] to hostname lwa8
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp1 10']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11'] to hostname lwa13
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp1 11']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12'] to hostname lwa26
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp1 12']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13'] to hostname lwa14
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp1 13']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14'] to hostname lwa22
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp1 14']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15'] to hostname lwa20
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp1 15']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16'] to hostname lwa1
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp1 16']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17'] to hostname lwa5
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp1 17']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18'] to hostname lwa25
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp1 18']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19'] to hostname lwa35
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp1 19']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20'] to hostname lwa19
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp1 20']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21'] to hostname lwa21
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp1 21']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22'] to hostname lwa6
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp1 22']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23'] to hostname lwa7
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp1 23']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24'] to hostname lwa33
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp1 24']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25'] to hostname lwa15
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp1 25']
2018-11-19 22:52:26 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26'] to hostname lwa28
2018-11-19 22:52:26 [DEBUG   ] spawn_process: at 2018-11-19 22:52:26 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp1 26']
2018-11-19 22:52:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:52:30 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:52:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:52:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:52:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:52:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:52:30 [CRITICAL] MGT underflow detected on beam(s) 1
2018-11-19 22:52:30 [DEBUG   ] Finished updating board flags in 1.209 seconds
2018-11-19 22:52:31 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 22:52:51 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:53:02 [DEBUG   ] Got command BAM from MCS: ref #563186
2018-11-19 22:53:02 [INFO    ] Finished processing BAM request
2018-11-19 22:53:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:53:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563186   8 58441 82382025 A  ERROR'
2018-11-19 22:53:02 [DEBUG   ] Got command BAM from MCS: ref #563187
2018-11-19 22:53:02 [INFO    ] Finished processing BAM request
2018-11-19 22:53:02 [DEBUG   ] bam = exited with status True, code 0x00
2018-11-19 22:53:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563187   8 58441 82382029 A  ERROR'
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1'] to hostname lwa27
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa27', '/lwa/software/bamx /lwa/test/bamx.tmp4 1']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2'] to hostname lwa24
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa24', '/lwa/software/bamx /lwa/test/bamx.tmp4 2']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3'] to hostname lwa23
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa23', '/lwa/software/bamx /lwa/test/bamx.tmp4 3']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4'] to hostname lwa18
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa18', '/lwa/software/bamx /lwa/test/bamx.tmp4 4']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5'] to hostname lwa10
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa10', '/lwa/software/bamx /lwa/test/bamx.tmp4 5']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6'] to hostname lwa32
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa32', '/lwa/software/bamx /lwa/test/bamx.tmp4 6']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7'] to hostname lwa11
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa11', '/lwa/software/bamx /lwa/test/bamx.tmp4 7']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8'] to hostname lwa30
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa30', '/lwa/software/bamx /lwa/test/bamx.tmp4 8']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9'] to hostname lwa12
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa12', '/lwa/software/bamx /lwa/test/bamx.tmp4 9']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10'] to hostname lwa8
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa8', '/lwa/software/bamx /lwa/test/bamx.tmp4 10']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11'] to hostname lwa13
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa13', '/lwa/software/bamx /lwa/test/bamx.tmp4 11']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12'] to hostname lwa26
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa26', '/lwa/software/bamx /lwa/test/bamx.tmp4 12']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13'] to hostname lwa14
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa14', '/lwa/software/bamx /lwa/test/bamx.tmp4 13']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14'] to hostname lwa22
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa22', '/lwa/software/bamx /lwa/test/bamx.tmp4 14']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15'] to hostname lwa20
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa20', '/lwa/software/bamx /lwa/test/bamx.tmp4 15']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16'] to hostname lwa1
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa1', '/lwa/software/bamx /lwa/test/bamx.tmp4 16']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17'] to hostname lwa5
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa5', '/lwa/software/bamx /lwa/test/bamx.tmp4 17']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18'] to hostname lwa25
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa25', '/lwa/software/bamx /lwa/test/bamx.tmp4 18']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19'] to hostname lwa35
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa35', '/lwa/software/bamx /lwa/test/bamx.tmp4 19']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20'] to hostname lwa19
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa19', '/lwa/software/bamx /lwa/test/bamx.tmp4 20']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21'] to hostname lwa21
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa21', '/lwa/software/bamx /lwa/test/bamx.tmp4 21']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22'] to hostname lwa6
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa6', '/lwa/software/bamx /lwa/test/bamx.tmp4 22']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23'] to hostname lwa7
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa7', '/lwa/software/bamx /lwa/test/bamx.tmp4 23']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24'] to hostname lwa33
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa33', '/lwa/software/bamx /lwa/test/bamx.tmp4 24']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25'] to hostname lwa15
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa15', '/lwa/software/bamx /lwa/test/bamx.tmp4 25']
2018-11-19 22:53:02 [INFO    ] SendBAM: Sent ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26'] to hostname lwa28
2018-11-19 22:53:02 [DEBUG   ] spawn_process: at 2018-11-19 22:53:02 running ['/usr/local/bin/lwa28', '/lwa/software/bamx /lwa/test/bamx.tmp4 26']
2018-11-19 22:53:11 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 22:53:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302362930352000000
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 22:53:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302362930352000000
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa27, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa27, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa27, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa27, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa24, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa24, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa24, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa23, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa23, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa23, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa23, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa18, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa18, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa18, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa18, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa10, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa10, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa11, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa11, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa11, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa11, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa30, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa30, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa30, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa30, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa12, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa12, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa12, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa12, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa8, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa8, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa8, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa13, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa13, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa13, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa13, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 5, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 5, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 5 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 1 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa14, chip 2, top port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 2 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 3 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa14, chip 4, bottom port
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 4 with 168174642
2018-11-19 22:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa22, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa20, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa20, chip 1, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa20, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa1, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa1, chip 1, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa1, chip 5, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa1, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa5, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa5, chip 1, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa5, chip 5, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa5, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa25, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa25, chip 1, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa25, chip 5, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa25, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa35, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa35, chip 1, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa35, chip 5, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa35, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa21, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa21, chip 1, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa21, chip 5, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa21, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa6, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa6, chip 1, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa6, chip 5, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa6, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa7, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa7, chip 5, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa7, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa33, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa33, chip 1, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa33, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa15, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa15, chip 1, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa15, chip 5, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa15, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa28, chip 1, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 1 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa28, chip 2, top port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 2 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 3 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa28, chip 4, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 4 with 168174642
2018-11-19 22:53:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa28, chip 5, bottom port
2018-11-19 22:53:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 5 with 168174642
2018-11-19 22:53:30 [WARNING ] (2, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (2, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (2, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (2, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (2, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (2, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (2, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (2, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (2, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (2, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (2, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (2, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (2, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (2, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (2, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (2, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (3, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (3, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (3, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (3, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (3, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (3, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (3, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (3, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (3, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (3, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (3, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (3, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (4, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (4, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (4, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (4, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (4, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (4, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (4, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (4, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (4, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (4, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (4, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (4, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (4, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (4, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (4, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (4, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (5, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (5, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (5, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (5, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (5, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (5, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (5, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (5, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (5, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (5, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (5, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (5, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (5, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (5, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (5, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (5, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (6, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (6, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (6, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (6, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (6, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (6, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (6, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (6, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (8, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (8, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (8, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (8, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (8, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (8, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (8, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (8, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (8, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (8, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (8, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (8, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (8, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (8, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (8, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (8, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (9, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (9, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (9, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (9, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (9, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (9, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (9, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (9, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (9, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (9, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (9, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (9, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (9, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (9, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (9, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (9, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (10, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (10, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (10, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (10, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (10, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (10, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (10, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (10, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (10, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (10, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (10, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (10, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (10, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (10, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (10, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (10, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (11, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (11, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (11, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (11, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (11, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (11, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (11, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (11, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (11, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (11, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (11, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (11, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (12, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (12, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (12, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (12, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (12, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (12, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (12, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (12, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (12, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (12, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (12, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (12, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (12, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (12, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (12, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (12, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (13, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (13, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (13, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (13, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (13, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (13, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (13, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (13, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (13, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (13, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (13, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (13, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (14, 2, 'top')
2018-11-19 22:53:30 [WARNING ] (14, 2, 'top')
2018-11-19 22:53:30 [WARNING ] (14, 2, 'top')
2018-11-19 22:53:30 [WARNING ] (14, 2, 'top')
2018-11-19 22:53:30 [WARNING ] (14, 4, 'bot')
2018-11-19 22:53:30 [WARNING ] (14, 4, 'bot')
2018-11-19 22:53:30 [WARNING ] (14, 4, 'bot')
2018-11-19 22:53:30 [WARNING ] (14, 4, 'bot')
2018-11-19 22:53:30 [WARNING ] (16, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (16, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (16, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (16, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (17, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (17, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (17, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (17, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (17, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (17, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (17, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (17, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (17, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (17, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (17, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (17, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (18, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (18, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (18, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (18, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (18, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (18, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (18, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (18, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (18, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (18, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (18, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (18, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (18, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (18, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (18, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (18, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (19, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (19, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (19, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (19, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (19, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (19, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (19, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (19, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (19, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (19, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (19, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (19, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (19, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (19, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (19, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (19, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (20, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (20, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (20, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (20, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (20, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (20, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (20, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (20, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (20, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (20, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (20, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (20, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (20, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (20, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (20, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (20, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (21, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (21, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (21, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (21, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (21, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (21, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (21, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (21, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (21, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (21, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (21, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (21, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (21, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (21, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (21, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (21, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (23, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (23, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (23, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (23, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (23, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (23, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (23, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (23, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (23, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (23, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (23, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (23, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (23, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (23, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (23, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (23, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (24, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (24, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (24, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (24, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (24, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (24, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (24, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (24, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (24, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (24, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (24, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (24, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (24, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (24, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (24, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (24, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (25, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (25, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (25, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (25, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (25, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (25, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (25, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (25, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (25, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (25, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (25, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (25, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (26, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (26, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (26, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (26, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (26, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (26, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (26, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (26, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (26, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (26, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (26, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (26, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (27, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (27, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (27, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (27, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (27, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (27, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (27, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (27, 1, 'bot')
2018-11-19 22:53:30 [WARNING ] (27, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (27, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (27, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (27, 5, 'top')
2018-11-19 22:53:30 [WARNING ] (27, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (27, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (27, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (27, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (28, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (28, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (28, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (28, 1, 'top')
2018-11-19 22:53:30 [WARNING ] (28, 2, 'top')
2018-11-19 22:53:30 [WARNING ] (28, 2, 'top')
2018-11-19 22:53:30 [WARNING ] (28, 2, 'top')
2018-11-19 22:53:30 [WARNING ] (28, 2, 'top')
2018-11-19 22:53:30 [WARNING ] (28, 4, 'bot')
2018-11-19 22:53:30 [WARNING ] (28, 4, 'bot')
2018-11-19 22:53:30 [WARNING ] (28, 4, 'bot')
2018-11-19 22:53:30 [WARNING ] (28, 4, 'bot')
2018-11-19 22:53:30 [WARNING ] (28, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (28, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (28, 5, 'bot')
2018-11-19 22:53:30 [WARNING ] (28, 5, 'bot')
2018-11-19 22:53:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 22:53:30 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 22:53:30 [DEBUG   ] Finished updating board flags in 1.195 seconds
2018-11-19 22:53:31 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 22:53:38 [DEBUG   ] Got command BAM from MCS: ref #563189
2018-11-19 22:53:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:53:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563189  75 58441 82418029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:53:38 [DEBUG   ] Got command BAM from MCS: ref #563188
2018-11-19 22:53:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:53:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563188  75 58441 82418030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:53:51 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-19 22:54:11 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:54:14 [DEBUG   ] Got command BAM from MCS: ref #563190
2018-11-19 22:54:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:54:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563190  75 58441 82454029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:54:14 [DEBUG   ] Got command BAM from MCS: ref #563191
2018-11-19 22:54:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:54:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563191  75 58441 82454030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa27, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa24, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa23, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa18, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa10, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa32, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa11, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa30, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa12, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa8, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa13, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa26, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa14, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 22:54:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302362942308000000
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 22:54:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302362942308000000
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa22, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa20, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa1, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa5, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa25, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa35, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa19, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa21, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 22:54:30 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa6, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa7, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa33, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa15, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 1 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 2 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 3 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 4 with 195999999
2018-11-19 22:54:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa28, chip 5 with 195999999
2018-11-19 22:54:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:54:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:54:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:54:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:54:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 22:54:30 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 22:54:30 [DEBUG   ] Finished updating board flags in 1.490 seconds
2018-11-19 22:54:31 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:54:50 [DEBUG   ] Got command BAM from MCS: ref #563193
2018-11-19 22:54:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:54:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563193  75 58441 82490029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:54:50 [DEBUG   ] Got command BAM from MCS: ref #563192
2018-11-19 22:54:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:54:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563192  75 58441 82490030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:54:51 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 22:55:02 [DEBUG   ] Got command RPT from MCS: ref #563194
2018-11-19 22:55:02 [DEBUG   ] summary = ERROR
2018-11-19 22:55:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563194  13 58441 82502173 A  ERRORERROR'
2018-11-19 22:55:02 [DEBUG   ] Got command RPT from MCS: ref #563195
2018-11-19 22:55:02 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 22:55:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563195  71 58441 82502377 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 22:55:02 [DEBUG   ] Got command RPT from MCS: ref #563196
2018-11-19 22:55:02 [DEBUG   ] summary = ERROR
2018-11-19 22:55:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563196  13 58441 82502569 A  ERRORERROR'
2018-11-19 22:55:02 [DEBUG   ] Got command RPT from MCS: ref #563197
2018-11-19 22:55:02 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 22:55:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563197  71 58441 82502773 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 22:55:02 [DEBUG   ] Got command RPT from MCS: ref #563198
2018-11-19 22:55:02 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 22:55:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563198  12 58441 82502977 A  ERRORB¤ff'
2018-11-19 22:55:11 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 22:55:26 [DEBUG   ] Got command BAM from MCS: ref #563203
2018-11-19 22:55:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:55:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563203  75 58441 82526029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:55:26 [DEBUG   ] Got command BAM from MCS: ref #563204
2018-11-19 22:55:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:55:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563204  75 58441 82526030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:55:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 22:55:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 22:55:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 22:55:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 22:55:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 22:55:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:55:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:55:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 22:55:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 22:55:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 22:55:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 22:55:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 22:55:31 [DEBUG   ] Finished updating ant stats in 0.430 seconds
2018-11-19 22:55:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:55:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:55:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:55:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:55:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:55:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 22:55:31 [DEBUG   ] Finished updating board flags in 1.974 seconds
2018-11-19 22:55:46 [DEBUG   ] Got command RPT from MCS: ref #563205
2018-11-19 22:55:46 [DEBUG   ] summary = ERROR
2018-11-19 22:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563205  13 58441 82546529 A  ERRORERROR'
2018-11-19 22:55:46 [DEBUG   ] Got command RPT from MCS: ref #563206
2018-11-19 22:55:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 22:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563206  98 58441 82546530 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 22:55:51 [DEBUG   ] Finished updating ant stats in 0.374 seconds
2018-11-19 22:56:02 [DEBUG   ] Got command BAM from MCS: ref #563223
2018-11-19 22:56:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:56:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563223  75 58441 82562025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:56:02 [DEBUG   ] Got command BAM from MCS: ref #563222
2018-11-19 22:56:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:56:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563222  75 58441 82562026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:56:12 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:56:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 22:56:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 22:56:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 22:56:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 22:56:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 22:56:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:56:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:56:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 22:56:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 22:56:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 22:56:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 22:56:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 22:56:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 22:56:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:56:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:56:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:56:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:56:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:56:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 22:56:31 [DEBUG   ] Finished updating board flags in 1.752 seconds
2018-11-19 22:56:32 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:56:38 [DEBUG   ] Got command BAM from MCS: ref #563225
2018-11-19 22:56:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:56:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563225  75 58441 82598021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:56:38 [DEBUG   ] Got command BAM from MCS: ref #563224
2018-11-19 22:56:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:56:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563224  75 58441 82598022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:56:52 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:57:12 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 22:57:14 [DEBUG   ] Got command BAM from MCS: ref #563226
2018-11-19 22:57:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:57:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563226  75 58441 82634021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:57:14 [DEBUG   ] Got command BAM from MCS: ref #563227
2018-11-19 22:57:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:57:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563227  75 58441 82634022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 22:57:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 22:57:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:57:30 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 22:57:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302362977588000000
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 22:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 22:57:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302362977588000000
2018-11-19 22:57:30 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:57:30 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:57:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:57:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:57:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:57:30 [WARNING ] (13, 1, 'bot')
2018-11-19 22:57:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 22:57:30 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 22:57:30 [DEBUG   ] Finished updating board flags in 0.997 seconds
2018-11-19 22:57:32 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-19 22:57:50 [DEBUG   ] Got command BAM from MCS: ref #563228
2018-11-19 22:57:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:57:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563228  75 58441 82670029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:57:50 [DEBUG   ] Got command BAM from MCS: ref #563229
2018-11-19 22:57:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:57:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563229  75 58441 82670030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:57:52 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:58:12 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 22:58:26 [DEBUG   ] Got command BAM from MCS: ref #563235
2018-11-19 22:58:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:58:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563235  75 58441 82706025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:58:26 [DEBUG   ] Got command BAM from MCS: ref #563234
2018-11-19 22:58:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:58:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563234  75 58441 82706026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 22:58:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302362989348000000
2018-11-19 22:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 22:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 22:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 22:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 22:58:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302362989348000000
2018-11-19 22:58:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 22:58:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:58:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 22:58:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 22:58:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 22:58:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 22:58:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 22:58:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:58:32 [DEBUG   ] Finished updating ant stats in 1.005 seconds
2018-11-19 22:58:32 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 22:58:32 [WARNING ] (13, 1, 'bot')
2018-11-19 22:58:32 [WARNING ] (13, 1, 'bot')
2018-11-19 22:58:32 [WARNING ] (13, 1, 'bot')
2018-11-19 22:58:32 [WARNING ] (13, 1, 'bot')
2018-11-19 22:58:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 22:58:32 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 22:58:32 [DEBUG   ] Finished updating board flags in 2.331 seconds
2018-11-19 22:58:52 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 22:59:02 [DEBUG   ] Got command BAM from MCS: ref #563237
2018-11-19 22:59:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:59:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563237  75 58441 82742025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:59:02 [DEBUG   ] Got command BAM from MCS: ref #563236
2018-11-19 22:59:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:59:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563236  75 58441 82742026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:59:12 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 22:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 22:59:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363001108000000
2018-11-19 22:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 22:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 22:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 22:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 22:59:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363001108000000
2018-11-19 22:59:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 22:59:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 22:59:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 22:59:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 22:59:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 22:59:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 22:59:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 22:59:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 22:59:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:59:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:59:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:59:31 [WARNING ] (13, 1, 'bot')
2018-11-19 22:59:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 22:59:31 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 22:59:31 [DEBUG   ] Finished updating board flags in 1.229 seconds
2018-11-19 22:59:32 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 22:59:38 [DEBUG   ] Got command BAM from MCS: ref #563238
2018-11-19 22:59:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:59:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563238  75 58441 82778025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:59:38 [DEBUG   ] Got command BAM from MCS: ref #563239
2018-11-19 22:59:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 22:59:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563239  75 58441 82778026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 22:59:52 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 23:00:02 [DEBUG   ] Got command RPT from MCS: ref #563242
2018-11-19 23:00:02 [DEBUG   ] summary = ERROR
2018-11-19 23:00:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563242  13 58441 82802257 A  ERRORERROR'
2018-11-19 23:00:02 [DEBUG   ] Got command RPT from MCS: ref #563243
2018-11-19 23:00:02 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:00:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563243  71 58441 82802461 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:00:02 [DEBUG   ] Got command RPT from MCS: ref #563244
2018-11-19 23:00:02 [DEBUG   ] summary = ERROR
2018-11-19 23:00:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563244  13 58441 82802665 A  ERRORERROR'
2018-11-19 23:00:02 [DEBUG   ] Got command RPT from MCS: ref #563246
2018-11-19 23:00:02 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:00:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563246  71 58441 82802857 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:00:03 [DEBUG   ] Got command RPT from MCS: ref #563248
2018-11-19 23:00:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:00:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563248  12 58441 82803061 A  ERRORB¤ff'
2018-11-19 23:00:12 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:00:14 [DEBUG   ] Got command BAM from MCS: ref #563253
2018-11-19 23:00:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:00:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563253  75 58441 82814025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:00:14 [DEBUG   ] Got command BAM from MCS: ref #563252
2018-11-19 23:00:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:00:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563252  75 58441 82814026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:00:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363012868000000
2018-11-19 23:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:00:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363012868000000
2018-11-19 23:00:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 23:00:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:00:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:00:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:00:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:00:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:00:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:00:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:00:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:00:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:00:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:00:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:00:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:00:31 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:00:31 [DEBUG   ] Finished updating board flags in 1.216 seconds
2018-11-19 23:00:32 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 23:00:46 [DEBUG   ] Got command RPT from MCS: ref #563254
2018-11-19 23:00:46 [DEBUG   ] summary = ERROR
2018-11-19 23:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563254  13 58441 82846529 A  ERRORERROR'
2018-11-19 23:00:46 [DEBUG   ] Got command RPT from MCS: ref #563255
2018-11-19 23:00:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563255  71 58441 82846530 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:00:50 [DEBUG   ] Got command BAM from MCS: ref #563271
2018-11-19 23:00:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:00:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563271  75 58441 82850021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:00:50 [DEBUG   ] Got command BAM from MCS: ref #563272
2018-11-19 23:00:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:00:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563272  75 58441 82850022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:00:52 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:01:12 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 23:01:26 [DEBUG   ] Got command BAM from MCS: ref #563278
2018-11-19 23:01:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:01:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563278  75 58441 82886021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:01:26 [DEBUG   ] Got command BAM from MCS: ref #563277
2018-11-19 23:01:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:01:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563277  75 58441 82886022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:01:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363024628000000
2018-11-19 23:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:01:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363024628000000
2018-11-19 23:01:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 23:01:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:01:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:01:30 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:01:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:01:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:01:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:01:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:01:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:01:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 23:01:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:01:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:01:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:01:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:01:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:01:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:01:31 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:01:31 [DEBUG   ] Finished updating board flags in 1.232 seconds
2018-11-19 23:01:32 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:01:52 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:02:02 [DEBUG   ] Got command BAM from MCS: ref #563279
2018-11-19 23:02:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:02:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563279  75 58441 82922029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:02:02 [DEBUG   ] Got command BAM from MCS: ref #563280
2018-11-19 23:02:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:02:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563280  75 58441 82922030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:02:12 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:02:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363036388000000
2018-11-19 23:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:02:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363036388000000
2018-11-19 23:02:30 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 23:02:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:02:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:02:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:02:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:02:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:02:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:02:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-19 23:02:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 23:02:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:02:31 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:02:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:02:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:02:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:02:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:02:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:02:31 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:02:31 [DEBUG   ] Finished updating board flags in 1.192 seconds
2018-11-19 23:02:32 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:02:38 [DEBUG   ] Got command BAM from MCS: ref #563281
2018-11-19 23:02:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:02:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563281  75 58441 82958025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:02:38 [DEBUG   ] Got command BAM from MCS: ref #563282
2018-11-19 23:02:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:02:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563282  75 58441 82958026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:02:52 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:03:12 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:03:14 [DEBUG   ] Got command BAM from MCS: ref #563283
2018-11-19 23:03:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:03:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563283  75 58441 82994021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:03:14 [DEBUG   ] Got command BAM from MCS: ref #563284
2018-11-19 23:03:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:03:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563284  75 58441 82994022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:03:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:03:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:03:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:03:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:03:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:03:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:03:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:03:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:03:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:03:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:03:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:03:32 [DEBUG   ] Finished updating ant stats in 0.629 seconds
2018-11-19 23:03:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:03:32 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:03:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:03:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:03:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:03:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:03:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:03:32 [DEBUG   ] Finished updating board flags in 2.062 seconds
2018-11-19 23:03:50 [DEBUG   ] Got command BAM from MCS: ref #563285
2018-11-19 23:03:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:03:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563285  75 58441 83030021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:03:50 [DEBUG   ] Got command BAM from MCS: ref #563286
2018-11-19 23:03:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:03:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563286  75 58441 83030022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:03:52 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-19 23:04:12 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:04:26 [DEBUG   ] Got command BAM from MCS: ref #563292
2018-11-19 23:04:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:04:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563292  75 58441 83066021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:04:26 [DEBUG   ] Got command BAM from MCS: ref #563291
2018-11-19 23:04:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:04:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563291  75 58441 83066022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:04:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:04:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363060104000000
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:04:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:04:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363060104000000
2018-11-19 23:04:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:04:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:04:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:04:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:04:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:04:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:04:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:04:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:04:31 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:04:31 [DEBUG   ] Finished updating board flags in 1.403 seconds
2018-11-19 23:04:32 [DEBUG   ] Finished updating ant stats in 0.399 seconds
2018-11-19 23:04:53 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:05:02 [DEBUG   ] Got command BAM from MCS: ref #563293
2018-11-19 23:05:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:05:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563293  75 58441 83102025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:05:02 [DEBUG   ] Got command BAM from MCS: ref #563294
2018-11-19 23:05:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:05:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563294  75 58441 83102026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:05:02 [DEBUG   ] Got command RPT from MCS: ref #563295
2018-11-19 23:05:02 [DEBUG   ] summary = ERROR
2018-11-19 23:05:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563295  13 58441 83102353 A  ERRORERROR'
2018-11-19 23:05:02 [DEBUG   ] Got command RPT from MCS: ref #563296
2018-11-19 23:05:02 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 23:05:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563296  71 58441 83102541 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 23:05:02 [DEBUG   ] Got command RPT from MCS: ref #563297
2018-11-19 23:05:02 [DEBUG   ] summary = ERROR
2018-11-19 23:05:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563297  13 58441 83102745 A  ERRORERROR'
2018-11-19 23:05:02 [DEBUG   ] Got command RPT from MCS: ref #563298
2018-11-19 23:05:02 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 23:05:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563298  71 58441 83102949 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 23:05:03 [DEBUG   ] Got command RPT from MCS: ref #563299
2018-11-19 23:05:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:05:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563299  12 58441 83103141 A  ERRORB¤ff'
2018-11-19 23:05:13 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:05:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:05:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363071864000000
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:05:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:05:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363071864000000
2018-11-19 23:05:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 23:05:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:05:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:05:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:05:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:05:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:05:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:05:31 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:05:31 [DEBUG   ] Finished updating board flags in 1.399 seconds
2018-11-19 23:05:33 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:05:38 [DEBUG   ] Got command BAM from MCS: ref #563301
2018-11-19 23:05:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:05:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563301  75 58441 83138029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:05:38 [DEBUG   ] Got command BAM from MCS: ref #563300
2018-11-19 23:05:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:05:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563300  75 58441 83138030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:05:46 [DEBUG   ] Got command RPT from MCS: ref #563302
2018-11-19 23:05:46 [DEBUG   ] summary = ERROR
2018-11-19 23:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563302  13 58441 83146525 A  ERRORERROR'
2018-11-19 23:05:46 [DEBUG   ] Got command RPT from MCS: ref #563303
2018-11-19 23:05:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 23:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563303  71 58441 83146526 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 23:05:53 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:06:13 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:06:14 [DEBUG   ] Got command BAM from MCS: ref #563320
2018-11-19 23:06:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:06:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563320  75 58441 83174021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:06:14 [DEBUG   ] Got command BAM from MCS: ref #563319
2018-11-19 23:06:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:06:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563319  75 58441 83174022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:06:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 23:06:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:06:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363083624000000
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:06:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:06:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363083624000000
2018-11-19 23:06:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:06:31 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:06:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:06:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:06:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:06:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:06:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:06:31 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:06:31 [DEBUG   ] Finished updating board flags in 1.223 seconds
2018-11-19 23:06:33 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:06:50 [DEBUG   ] Got command BAM from MCS: ref #563322
2018-11-19 23:06:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:06:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563322  75 58441 83210029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:06:50 [DEBUG   ] Got command BAM from MCS: ref #563321
2018-11-19 23:06:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:06:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563321  75 58441 83210030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:06:53 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:07:13 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:07:26 [DEBUG   ] Got command BAM from MCS: ref #563328
2018-11-19 23:07:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:07:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563328  75 58441 83246025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:07:26 [DEBUG   ] Got command BAM from MCS: ref #563327
2018-11-19 23:07:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:07:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563327  75 58441 83246026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:07:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:07:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:07:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:07:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:07:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:07:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:07:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:07:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:07:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:07:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:07:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:07:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:07:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:07:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:07:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:07:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:07:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:07:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:07:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:07:32 [DEBUG   ] Finished updating board flags in 1.734 seconds
2018-11-19 23:07:33 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:07:53 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:08:02 [DEBUG   ] Got command BAM from MCS: ref #563329
2018-11-19 23:08:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:08:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563329  75 58441 83282021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:08:02 [DEBUG   ] Got command BAM from MCS: ref #563330
2018-11-19 23:08:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:08:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563330  75 58441 83282022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:08:13 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:08:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:08:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363107144000000
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:08:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:08:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363107144000000
2018-11-19 23:08:31 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:08:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:08:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:08:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:08:31 [WARNING ] (13, 1, 'bot')
2018-11-19 23:08:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:08:31 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:08:31 [DEBUG   ] Finished updating board flags in 1.007 seconds
2018-11-19 23:08:33 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:08:38 [DEBUG   ] Got command BAM from MCS: ref #563331
2018-11-19 23:08:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:08:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563331  75 58441 83318029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:08:38 [DEBUG   ] Got command BAM from MCS: ref #563332
2018-11-19 23:08:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:08:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563332  75 58441 83318030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:08:53 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:09:13 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:09:14 [DEBUG   ] Got command BAM from MCS: ref #563333
2018-11-19 23:09:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:09:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563333  75 58441 83354021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:09:14 [DEBUG   ] Got command BAM from MCS: ref #563334
2018-11-19 23:09:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:09:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563334  75 58441 83354022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:09:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:09:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:09:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:09:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:09:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:09:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:09:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:09:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:09:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:09:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:09:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:09:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:09:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:09:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:09:32 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:09:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:09:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:09:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:09:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:09:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:09:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:09:32 [DEBUG   ] Finished updating board flags in 1.599 seconds
2018-11-19 23:09:33 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:09:50 [DEBUG   ] Got command BAM from MCS: ref #563336
2018-11-19 23:09:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:09:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563336  75 58441 83390025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:09:50 [DEBUG   ] Got command BAM from MCS: ref #563335
2018-11-19 23:09:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:09:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563335  75 58441 83390026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:09:53 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:10:02 [DEBUG   ] Got command RPT from MCS: ref #563339
2018-11-19 23:10:02 [DEBUG   ] summary = ERROR
2018-11-19 23:10:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563339  13 58441 83402429 A  ERRORERROR'
2018-11-19 23:10:02 [DEBUG   ] Got command RPT from MCS: ref #563340
2018-11-19 23:10:02 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 23:10:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563340  98 58441 83402633 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 23:10:02 [DEBUG   ] Got command RPT from MCS: ref #563341
2018-11-19 23:10:02 [DEBUG   ] summary = ERROR
2018-11-19 23:10:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563341  13 58441 83402825 A  ERRORERROR'
2018-11-19 23:10:03 [DEBUG   ] Got command RPT from MCS: ref #563343
2018-11-19 23:10:03 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 23:10:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563343  98 58441 83403029 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 23:10:03 [DEBUG   ] Got command RPT from MCS: ref #563345
2018-11-19 23:10:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:10:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563345  12 58441 83403233 A  ERRORB¤ff'
2018-11-19 23:10:13 [DEBUG   ] Finished updating ant stats in 1.194 seconds
2018-11-19 23:10:26 [DEBUG   ] Got command BAM from MCS: ref #563353
2018-11-19 23:10:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:10:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563353  75 58441 83426029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:10:26 [DEBUG   ] Got command BAM from MCS: ref #563354
2018-11-19 23:10:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:10:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563354  75 58441 83426030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:10:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363130664000000
2018-11-19 23:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:10:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363130664000000
2018-11-19 23:10:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:10:31 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:10:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:10:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:10:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:10:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:10:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:10:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:10:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:10:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:10:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:10:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:10:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:10:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:10:32 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:10:32 [DEBUG   ] Finished updating board flags in 1.399 seconds
2018-11-19 23:10:33 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:10:46 [DEBUG   ] Got command RPT from MCS: ref #563355
2018-11-19 23:10:46 [DEBUG   ] summary = ERROR
2018-11-19 23:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563355  13 58441 83446529 A  ERRORERROR'
2018-11-19 23:10:46 [DEBUG   ] Got command RPT from MCS: ref #563356
2018-11-19 23:10:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563356  71 58441 83446530 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:10:53 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:11:02 [DEBUG   ] Got command BAM from MCS: ref #563373
2018-11-19 23:11:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:11:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563373  75 58441 83462025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:11:02 [DEBUG   ] Got command BAM from MCS: ref #563372
2018-11-19 23:11:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:11:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563372  75 58441 83462026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:11:13 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:11:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363142424000000
2018-11-19 23:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:11:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363142424000000
2018-11-19 23:11:31 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 23:11:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:11:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:11:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:11:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:11:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:11:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:11:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:11:32 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:11:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:11:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:11:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:11:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:11:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:11:32 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:11:32 [DEBUG   ] Finished updating board flags in 1.220 seconds
2018-11-19 23:11:33 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:11:38 [DEBUG   ] Got command BAM from MCS: ref #563374
2018-11-19 23:11:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:11:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563374  75 58441 83498029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:11:38 [DEBUG   ] Got command BAM from MCS: ref #563375
2018-11-19 23:11:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:11:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563375  75 58441 83498030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:11:53 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:12:13 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:12:14 [DEBUG   ] Got command BAM from MCS: ref #563376
2018-11-19 23:12:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:12:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563376  75 58441 83534021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:12:14 [DEBUG   ] Got command BAM from MCS: ref #563377
2018-11-19 23:12:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:12:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563377  75 58441 83534022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:12:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363154184000000
2018-11-19 23:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:12:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363154184000000
2018-11-19 23:12:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:12:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:12:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:12:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:12:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:12:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:12:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:12:32 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:12:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:12:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:12:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:12:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:12:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:12:32 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:12:32 [DEBUG   ] Finished updating board flags in 1.185 seconds
2018-11-19 23:12:33 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:12:50 [DEBUG   ] Got command BAM from MCS: ref #563378
2018-11-19 23:12:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:12:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563378  75 58441 83570029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:12:50 [DEBUG   ] Got command BAM from MCS: ref #563379
2018-11-19 23:12:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:12:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563379  75 58441 83570030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:12:53 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:13:13 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:13:26 [DEBUG   ] Got command BAM from MCS: ref #563385
2018-11-19 23:13:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:13:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563385  75 58441 83606021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:13:26 [DEBUG   ] Got command BAM from MCS: ref #563384
2018-11-19 23:13:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:13:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563384  75 58441 83606022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:13:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:13:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:13:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:13:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:13:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:13:33 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 23:13:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:13:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 23:13:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:13:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363166336000000
2018-11-19 23:13:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:13:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:13:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:13:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:13:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363166336000000
2018-11-19 23:13:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:13:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:13:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:13:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:13:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:13:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:13:33 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:13:33 [DEBUG   ] Finished updating board flags in 2.473 seconds
2018-11-19 23:13:53 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 23:14:02 [DEBUG   ] Got command BAM from MCS: ref #563387
2018-11-19 23:14:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:14:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563387  75 58441 83642025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:14:02 [DEBUG   ] Got command BAM from MCS: ref #563386
2018-11-19 23:14:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:14:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563386  75 58441 83642026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:14:13 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:14:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:14:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:14:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:14:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:14:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:14:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:14:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:14:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:14:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:14:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:14:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:14:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:14:33 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-19 23:14:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:14:33 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:14:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:14:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:14:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:14:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:14:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:14:33 [DEBUG   ] Finished updating board flags in 2.102 seconds
2018-11-19 23:14:38 [DEBUG   ] Got command BAM from MCS: ref #563389
2018-11-19 23:14:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:14:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563389  75 58441 83678025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:14:38 [DEBUG   ] Got command BAM from MCS: ref #563388
2018-11-19 23:14:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:14:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563388  75 58441 83678026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:14:53 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-19 23:15:02 [DEBUG   ] Got command RPT from MCS: ref #563390
2018-11-19 23:15:02 [DEBUG   ] summary = ERROR
2018-11-19 23:15:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563390  13 58441 83702521 A  ERRORERROR'
2018-11-19 23:15:02 [DEBUG   ] Got command RPT from MCS: ref #563391
2018-11-19 23:15:02 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 23:15:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563391  98 58441 83702713 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 23:15:02 [DEBUG   ] Got command RPT from MCS: ref #563392
2018-11-19 23:15:02 [DEBUG   ] summary = ERROR
2018-11-19 23:15:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563392  13 58441 83702917 A  ERRORERROR'
2018-11-19 23:15:03 [DEBUG   ] Got command RPT from MCS: ref #563393
2018-11-19 23:15:03 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 23:15:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563393  98 58441 83703121 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 23:15:03 [DEBUG   ] Got command RPT from MCS: ref #563394
2018-11-19 23:15:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:15:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563394  12 58441 83703313 A  ERRORB¤ff'
2018-11-19 23:15:13 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:15:14 [DEBUG   ] Got command BAM from MCS: ref #563396
2018-11-19 23:15:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:15:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563396  75 58441 83714025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:15:14 [DEBUG   ] Got command BAM from MCS: ref #563395
2018-11-19 23:15:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:15:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563395  75 58441 83714026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:15:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:15:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:15:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:15:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:15:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:15:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 23:15:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:15:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 23:15:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:15:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:15:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:15:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:15:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:15:33 [DEBUG   ] Finished updating ant stats in 0.629 seconds
2018-11-19 23:15:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:15:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:15:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:15:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:15:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:15:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:15:33 [DEBUG   ] Finished updating board flags in 1.855 seconds
2018-11-19 23:15:46 [DEBUG   ] Got command RPT from MCS: ref #563397
2018-11-19 23:15:46 [DEBUG   ] summary = ERROR
2018-11-19 23:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563397  13 58441 83746533 A  ERRORERROR'
2018-11-19 23:15:46 [DEBUG   ] Got command RPT from MCS: ref #563398
2018-11-19 23:15:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 23:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563398  98 58441 83746534 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 23:15:50 [DEBUG   ] Got command BAM from MCS: ref #563414
2018-11-19 23:15:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:15:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563414  75 58441 83750025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:15:50 [DEBUG   ] Got command BAM from MCS: ref #563415
2018-11-19 23:15:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:15:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563415  75 58441 83750026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:15:53 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-19 23:16:13 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:16:26 [DEBUG   ] Got command BAM from MCS: ref #563420
2018-11-19 23:16:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:16:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563420  75 58441 83786025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:16:26 [DEBUG   ] Got command BAM from MCS: ref #563421
2018-11-19 23:16:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:16:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563421  75 58441 83786026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:16:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 23:16:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:16:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363201420000000
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:16:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:16:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363201420000000
2018-11-19 23:16:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 23:16:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:16:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:16:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:16:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:16:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:16:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:16:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:16:32 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:16:32 [DEBUG   ] Finished updating board flags in 1.177 seconds
2018-11-19 23:16:33 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:16:53 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 23:17:02 [DEBUG   ] Got command BAM from MCS: ref #563422
2018-11-19 23:17:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:17:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563422  75 58441 83822021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:17:02 [DEBUG   ] Got command BAM from MCS: ref #563423
2018-11-19 23:17:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:17:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563423  75 58441 83822022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:17:13 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:17:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 23:17:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:17:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363213180000000
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:17:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:17:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363213180000000
2018-11-19 23:17:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 23:17:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:17:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:17:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:17:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:17:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:17:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:17:32 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:17:32 [DEBUG   ] Finished updating board flags in 1.069 seconds
2018-11-19 23:17:33 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:17:38 [DEBUG   ] Got command BAM from MCS: ref #563424
2018-11-19 23:17:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:17:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563424  75 58441 83858021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:17:38 [DEBUG   ] Got command BAM from MCS: ref #563425
2018-11-19 23:17:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:17:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563425  75 58441 83858022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:17:53 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:18:13 [DEBUG   ] Finished updating ant stats in 0.394 seconds
2018-11-19 23:18:14 [DEBUG   ] Got command BAM from MCS: ref #563426
2018-11-19 23:18:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:18:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563426  75 58441 83894021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:18:14 [DEBUG   ] Got command BAM from MCS: ref #563427
2018-11-19 23:18:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:18:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563427  75 58441 83894022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:18:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 23:18:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:18:32 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:18:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363224940000000
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:18:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:18:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363224940000000
2018-11-19 23:18:32 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:18:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:18:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:18:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:18:32 [WARNING ] (13, 1, 'bot')
2018-11-19 23:18:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:18:32 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:18:32 [DEBUG   ] Finished updating board flags in 0.932 seconds
2018-11-19 23:18:34 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:18:50 [DEBUG   ] Got command BAM from MCS: ref #563429
2018-11-19 23:18:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:18:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563429  75 58441 83930021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:18:50 [DEBUG   ] Got command BAM from MCS: ref #563428
2018-11-19 23:18:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:18:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563428  75 58441 83930022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:18:54 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:19:14 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:19:26 [DEBUG   ] Got command BAM from MCS: ref #563434
2018-11-19 23:19:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:19:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563434  75 58441 83966029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:19:26 [DEBUG   ] Got command BAM from MCS: ref #563435
2018-11-19 23:19:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:19:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563435  75 58441 83966030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:19:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:19:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363236700000000
2018-11-19 23:19:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:19:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:19:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:19:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:19:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363236700000000
2018-11-19 23:19:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:19:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:19:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:19:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:19:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:19:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:19:34 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 23:19:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-19 23:19:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:19:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:19:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:19:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:19:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:19:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:19:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:19:34 [DEBUG   ] Finished updating board flags in 2.548 seconds
2018-11-19 23:19:54 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 23:20:02 [DEBUG   ] Got command RPT from MCS: ref #563440
2018-11-19 23:20:02 [DEBUG   ] summary = ERROR
2018-11-19 23:20:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563440  13 58441 84002605 A  ERRORERROR'
2018-11-19 23:20:02 [DEBUG   ] Got command RPT from MCS: ref #563441
2018-11-19 23:20:02 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:20:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563441  71 58441 84002809 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:20:03 [DEBUG   ] Got command RPT from MCS: ref #563442
2018-11-19 23:20:03 [DEBUG   ] summary = ERROR
2018-11-19 23:20:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563442  13 58441 84003001 A  ERRORERROR'
2018-11-19 23:20:03 [DEBUG   ] Got command BAM from MCS: ref #563438
2018-11-19 23:20:03 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:20:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563438  75 58441 84003025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:20:03 [DEBUG   ] Got command BAM from MCS: ref #563439
2018-11-19 23:20:03 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:20:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563439  75 58441 84003026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:20:03 [DEBUG   ] Got command RPT from MCS: ref #563444
2018-11-19 23:20:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:20:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563444  71 58441 84003205 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:20:03 [DEBUG   ] Got command RPT from MCS: ref #563446
2018-11-19 23:20:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:20:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563446  12 58441 84003409 A  ERRORB¤ff'
2018-11-19 23:20:14 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:20:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:20:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363248460000000
2018-11-19 23:20:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:20:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:20:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:20:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:20:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363248460000000
2018-11-19 23:20:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:20:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:20:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:20:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:20:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:20:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:20:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:20:34 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 23:20:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:20:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:20:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:20:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:20:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:20:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:20:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:20:34 [DEBUG   ] Finished updating board flags in 2.154 seconds
2018-11-19 23:20:40 [DEBUG   ] Got command BAM from MCS: ref #563450
2018-11-19 23:20:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:20:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563450  75 58441 84040021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:20:40 [DEBUG   ] Got command BAM from MCS: ref #563451
2018-11-19 23:20:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:20:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563451  75 58441 84040022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:20:46 [DEBUG   ] Got command RPT from MCS: ref #563452
2018-11-19 23:20:46 [DEBUG   ] summary = ERROR
2018-11-19 23:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563452  13 58441 84046537 A  ERRORERROR'
2018-11-19 23:20:46 [DEBUG   ] Got command RPT from MCS: ref #563453
2018-11-19 23:20:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563453  71 58441 84046538 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:20:54 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-19 23:21:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:21:17 [DEBUG   ] Got command BAM from MCS: ref #563470
2018-11-19 23:21:17 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:21:17 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563470  75 58441 84077021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:21:17 [DEBUG   ] Got command BAM from MCS: ref #563469
2018-11-19 23:21:17 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:21:17 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563469  75 58441 84077022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:21:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:21:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363260220000000
2018-11-19 23:21:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:21:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:21:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:21:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:21:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363260220000000
2018-11-19 23:21:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 23:21:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 23:21:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:21:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:21:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:21:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:21:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:21:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:21:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:21:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:21:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:21:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:21:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:21:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:21:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:21:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:21:33 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:21:33 [DEBUG   ] Finished updating board flags in 1.090 seconds
2018-11-19 23:21:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 23:21:54 [DEBUG   ] Got command BAM from MCS: ref #563471
2018-11-19 23:21:54 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:21:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563471  75 58441 84114029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:21:54 [DEBUG   ] Got command BAM from MCS: ref #563472
2018-11-19 23:21:54 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:21:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563472  75 58441 84114030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:21:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:22:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:22:31 [DEBUG   ] Got command BAM from MCS: ref #563478
2018-11-19 23:22:31 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:22:31 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563478  75 58441 84151025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:22:31 [DEBUG   ] Got command BAM from MCS: ref #563477
2018-11-19 23:22:31 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:22:31 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563477  75 58441 84151026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:22:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:22:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:22:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363272176000000
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:22:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:22:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363272176000000
2018-11-19 23:22:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:22:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:22:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:22:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:22:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:22:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:22:33 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:22:33 [DEBUG   ] Finished updating board flags in 1.479 seconds
2018-11-19 23:22:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:22:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:23:08 [DEBUG   ] Got command BAM from MCS: ref #563479
2018-11-19 23:23:08 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:23:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563479  75 58441 84188021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:23:08 [DEBUG   ] Got command BAM from MCS: ref #563480
2018-11-19 23:23:08 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:23:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563480  75 58441 84188022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:23:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:23:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:23:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363283936000000
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:23:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:23:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363283936000000
2018-11-19 23:23:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:23:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:23:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:23:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:23:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:23:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:23:33 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:23:33 [DEBUG   ] Finished updating board flags in 1.375 seconds
2018-11-19 23:23:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:23:45 [DEBUG   ] Got command BAM from MCS: ref #563482
2018-11-19 23:23:45 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:23:45 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563482  75 58441 84225025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:23:45 [DEBUG   ] Got command BAM from MCS: ref #563481
2018-11-19 23:23:45 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:23:45 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563481  75 58441 84225026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:23:54 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:24:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:24:22 [DEBUG   ] Got command BAM from MCS: ref #563483
2018-11-19 23:24:22 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:24:22 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563483  75 58441 84262025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:24:22 [DEBUG   ] Got command BAM from MCS: ref #563484
2018-11-19 23:24:22 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:24:22 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563484  75 58441 84262026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:24:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:24:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:24:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:24:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:24:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:24:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:24:34 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-19 23:24:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:24:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363295892000000
2018-11-19 23:24:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:24:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:24:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:24:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:24:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363295892000000
2018-11-19 23:24:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:24:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:24:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:24:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:24:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:24:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:24:34 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:24:34 [DEBUG   ] Finished updating board flags in 2.007 seconds
2018-11-19 23:24:54 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-19 23:24:59 [DEBUG   ] Got command BAM from MCS: ref #563485
2018-11-19 23:24:59 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:24:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563485  75 58441 84299021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:24:59 [DEBUG   ] Got command BAM from MCS: ref #563486
2018-11-19 23:24:59 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:24:59 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563486  75 58441 84299022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:25:02 [DEBUG   ] Got command RPT from MCS: ref #563487
2018-11-19 23:25:02 [DEBUG   ] summary = ERROR
2018-11-19 23:25:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563487  13 58441 84302693 A  ERRORERROR'
2018-11-19 23:25:02 [DEBUG   ] Got command RPT from MCS: ref #563488
2018-11-19 23:25:02 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 23:25:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563488  71 58441 84302885 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 23:25:03 [DEBUG   ] Got command RPT from MCS: ref #563489
2018-11-19 23:25:03 [DEBUG   ] summary = ERROR
2018-11-19 23:25:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563489  13 58441 84303089 A  ERRORERROR'
2018-11-19 23:25:03 [DEBUG   ] Got command RPT from MCS: ref #563490
2018-11-19 23:25:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 23:25:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563490  71 58441 84303293 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 23:25:03 [DEBUG   ] Got command RPT from MCS: ref #563491
2018-11-19 23:25:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:25:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563491  12 58441 84303497 A  ERRORB¤ff'
2018-11-19 23:25:14 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:25:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:25:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:25:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:25:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:25:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:25:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 23:25:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:25:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:25:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:25:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:25:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:25:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:25:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:25:34 [DEBUG   ] Finished updating ant stats in 0.628 seconds
2018-11-19 23:25:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:25:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:25:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:25:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:25:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:25:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:25:34 [DEBUG   ] Finished updating board flags in 1.790 seconds
2018-11-19 23:25:36 [DEBUG   ] Got command BAM from MCS: ref #563497
2018-11-19 23:25:36 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:25:36 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563497  75 58441 84336025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:25:36 [DEBUG   ] Got command BAM from MCS: ref #563496
2018-11-19 23:25:36 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:25:36 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563496  75 58441 84336026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:25:46 [DEBUG   ] Got command RPT from MCS: ref #563498
2018-11-19 23:25:46 [DEBUG   ] summary = ERROR
2018-11-19 23:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563498  13 58441 84346537 A  ERRORERROR'
2018-11-19 23:25:46 [DEBUG   ] Got command RPT from MCS: ref #563499
2018-11-19 23:25:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 23:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563499  98 58441 84346538 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 23:25:54 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-19 23:26:13 [DEBUG   ] Got command BAM from MCS: ref #563515
2018-11-19 23:26:13 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:26:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563515  75 58441 84373029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:26:13 [DEBUG   ] Got command BAM from MCS: ref #563516
2018-11-19 23:26:13 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:26:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563516  75 58441 84373030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:26:15 [DEBUG   ] Finished updating ant stats in 1.402 seconds
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:26:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:26:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:26:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363319216000000
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:26:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:26:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363319216000000
2018-11-19 23:26:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 23:26:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:26:33 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:26:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:26:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:26:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:26:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:26:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:26:33 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:26:33 [DEBUG   ] Finished updating board flags in 1.056 seconds
2018-11-19 23:26:34 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:26:50 [DEBUG   ] Got command BAM from MCS: ref #563518
2018-11-19 23:26:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:26:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563518  75 58441 84410029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:26:50 [DEBUG   ] Got command BAM from MCS: ref #563517
2018-11-19 23:26:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:26:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563517  75 58441 84410030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:26:55 [DEBUG   ] Finished updating ant stats in 1.402 seconds
2018-11-19 23:27:14 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:27:27 [DEBUG   ] Got command BAM from MCS: ref #563519
2018-11-19 23:27:27 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:27:27 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563519  75 58441 84447029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:27:27 [DEBUG   ] Got command BAM from MCS: ref #563520
2018-11-19 23:27:27 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:27:27 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563520  75 58441 84447030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:27:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 23:27:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 23:27:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:27:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363330976000000
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:27:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:27:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363330976000000
2018-11-19 23:27:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:27:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:27:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:27:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:27:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:27:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:27:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:27:33 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:27:33 [DEBUG   ] Finished updating board flags in 1.017 seconds
2018-11-19 23:27:34 [DEBUG   ] Finished updating ant stats in 0.398 seconds
2018-11-19 23:27:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:28:04 [DEBUG   ] Got command BAM from MCS: ref #563521
2018-11-19 23:28:04 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:28:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563521  75 58441 84484025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:28:04 [DEBUG   ] Got command BAM from MCS: ref #563522
2018-11-19 23:28:04 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:28:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563522  75 58441 84484026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:28:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:28:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:28:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363342736000000
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:28:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:28:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363342736000000
2018-11-19 23:28:33 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:28:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:28:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:28:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:28:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:28:33 [WARNING ] (13, 1, 'bot')
2018-11-19 23:28:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:28:33 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:28:33 [DEBUG   ] Finished updating board flags in 0.902 seconds
2018-11-19 23:28:35 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:28:41 [DEBUG   ] Got command BAM from MCS: ref #563528
2018-11-19 23:28:41 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:28:41 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563528  75 58441 84521029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:28:41 [DEBUG   ] Got command BAM from MCS: ref #563527
2018-11-19 23:28:41 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:28:41 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563527  75 58441 84521030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:28:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:29:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:29:18 [DEBUG   ] Got command BAM from MCS: ref #563529
2018-11-19 23:29:18 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:29:18 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563529  75 58441 84558025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:29:18 [DEBUG   ] Got command BAM from MCS: ref #563530
2018-11-19 23:29:18 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:29:18 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563530  75 58441 84558026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:29:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:29:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363354496000000
2018-11-19 23:29:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:29:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:29:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:29:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:29:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363354496000000
2018-11-19 23:29:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:29:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:29:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:29:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:29:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:29:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:29:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:29:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:29:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:29:34 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:29:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:29:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:29:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:29:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:29:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:29:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:29:34 [DEBUG   ] Finished updating board flags in 1.406 seconds
2018-11-19 23:29:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:29:55 [DEBUG   ] Got command BAM from MCS: ref #563533
2018-11-19 23:29:55 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:29:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563533  75 58441 84595030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:29:55 [DEBUG   ] Got command BAM from MCS: ref #563534
2018-11-19 23:29:55 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:29:55 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563534  75 58441 84595030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:29:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:30:02 [DEBUG   ] Got command RPT from MCS: ref #563535
2018-11-19 23:30:02 [DEBUG   ] summary = ERROR
2018-11-19 23:30:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563535  13 58441 84602773 A  ERRORERROR'
2018-11-19 23:30:02 [DEBUG   ] Got command RPT from MCS: ref #563536
2018-11-19 23:30:02 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:30:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563536  71 58441 84602977 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:30:03 [DEBUG   ] Got command RPT from MCS: ref #563537
2018-11-19 23:30:03 [DEBUG   ] summary = ERROR
2018-11-19 23:30:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563537  13 58441 84603169 A  ERRORERROR'
2018-11-19 23:30:03 [DEBUG   ] Got command RPT from MCS: ref #563538
2018-11-19 23:30:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:30:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563538  71 58441 84603373 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:30:03 [DEBUG   ] Got command RPT from MCS: ref #563540
2018-11-19 23:30:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:30:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563540  12 58441 84603577 A  ERRORB¤ff'
2018-11-19 23:30:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:30:32 [DEBUG   ] Got command BAM from MCS: ref #563546
2018-11-19 23:30:32 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:30:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563546  75 58441 84632029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:30:32 [DEBUG   ] Got command BAM from MCS: ref #563545
2018-11-19 23:30:32 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:30:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563545  75 58441 84632030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:30:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:30:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363366256000000
2018-11-19 23:30:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:30:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:30:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:30:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:30:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363366256000000
2018-11-19 23:30:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 23:30:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:30:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:30:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:30:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:30:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:30:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:30:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:30:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:30:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:30:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:30:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:30:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:30:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:30:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:30:34 [DEBUG   ] Finished updating board flags in 1.434 seconds
2018-11-19 23:30:35 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:30:46 [DEBUG   ] Got command RPT from MCS: ref #563547
2018-11-19 23:30:46 [DEBUG   ] summary = ERROR
2018-11-19 23:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563547  13 58441 84646537 A  ERRORERROR'
2018-11-19 23:30:46 [DEBUG   ] Got command RPT from MCS: ref #563548
2018-11-19 23:30:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563548  71 58441 84646538 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:30:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:31:09 [DEBUG   ] Got command BAM from MCS: ref #563565
2018-11-19 23:31:09 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:31:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563565  75 58441 84669021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:31:09 [DEBUG   ] Got command BAM from MCS: ref #563564
2018-11-19 23:31:09 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:31:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563564  75 58441 84669022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:31:15 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:31:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:31:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363378016000000
2018-11-19 23:31:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:31:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:31:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:31:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:31:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363378016000000
2018-11-19 23:31:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 23:31:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:31:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:31:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:31:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:31:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:31:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:31:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:31:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 23:31:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:31:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:31:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:31:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:31:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:31:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:31:34 [DEBUG   ] Finished updating board flags in 1.438 seconds
2018-11-19 23:31:35 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:31:46 [DEBUG   ] Got command BAM from MCS: ref #563570
2018-11-19 23:31:46 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:31:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563570  75 58441 84706025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:31:46 [DEBUG   ] Got command BAM from MCS: ref #563571
2018-11-19 23:31:46 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:31:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563571  75 58441 84706026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:31:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:32:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:32:23 [DEBUG   ] Got command BAM from MCS: ref #563572
2018-11-19 23:32:23 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:32:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563572  75 58441 84743021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:32:23 [DEBUG   ] Got command BAM from MCS: ref #563573
2018-11-19 23:32:23 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:32:23 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563573  75 58441 84743022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:32:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363389776000000
2018-11-19 23:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:32:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363389776000000
2018-11-19 23:32:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 23:32:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:32:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:32:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:32:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:32:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:32:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:32:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:32:35 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-19 23:32:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:32:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:32:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:32:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:32:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:32:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:32:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:32:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:32:35 [DEBUG   ] Finished updating board flags in 2.526 seconds
2018-11-19 23:32:55 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-19 23:33:00 [DEBUG   ] Got command BAM from MCS: ref #563574
2018-11-19 23:33:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:33:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563574  75 58441 84780029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:33:00 [DEBUG   ] Got command BAM from MCS: ref #563575
2018-11-19 23:33:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:33:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563575  75 58441 84780030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:33:15 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-19 23:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:33:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363401536000000
2018-11-19 23:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:33:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363401536000000
2018-11-19 23:33:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:33:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:33:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:33:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363401732000000
2018-11-19 23:33:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:33:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:33:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:33:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:33:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363401732000000
2018-11-19 23:33:35 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-19 23:33:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:33:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:33:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:33:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:33:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:33:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:33:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-19 23:33:35 [DEBUG   ] Finished updating board flags in 2.328 seconds
2018-11-19 23:33:37 [DEBUG   ] Got command BAM from MCS: ref #563577
2018-11-19 23:33:37 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:33:37 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563577  75 58441 84817029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:33:37 [DEBUG   ] Got command BAM from MCS: ref #563576
2018-11-19 23:33:37 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:33:37 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563576  75 58441 84817030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:33:55 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-19 23:34:14 [DEBUG   ] Got command BAM from MCS: ref #563582
2018-11-19 23:34:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:34:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563582  75 58441 84854025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:34:14 [DEBUG   ] Got command BAM from MCS: ref #563583
2018-11-19 23:34:14 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:34:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563583  75 58441 84854026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:34:15 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:34:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363413296000000
2018-11-19 23:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:34:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363413296000000
2018-11-19 23:34:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:34:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:34:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:34:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:34:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:34:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:34:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:34:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:34:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:34:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:34:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:34:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:34:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:34:34 [DEBUG   ] Finished updating board flags in 1.045 seconds
2018-11-19 23:34:35 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:34:51 [DEBUG   ] Got command BAM from MCS: ref #563585
2018-11-19 23:34:51 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:34:51 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563585  75 58441 84891029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:34:51 [DEBUG   ] Got command BAM from MCS: ref #563584
2018-11-19 23:34:51 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:34:51 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563584  75 58441 84891030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:34:55 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:35:02 [DEBUG   ] Got command RPT from MCS: ref #563586
2018-11-19 23:35:02 [DEBUG   ] summary = ERROR
2018-11-19 23:35:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563586  13 58441 84902861 A  ERRORERROR'
2018-11-19 23:35:03 [DEBUG   ] Got command RPT from MCS: ref #563587
2018-11-19 23:35:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:35:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563587  71 58441 84903053 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:35:03 [DEBUG   ] Got command RPT from MCS: ref #563588
2018-11-19 23:35:03 [DEBUG   ] summary = ERROR
2018-11-19 23:35:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563588  13 58441 84903257 A  ERRORERROR'
2018-11-19 23:35:03 [DEBUG   ] Got command RPT from MCS: ref #563589
2018-11-19 23:35:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:35:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563589  71 58441 84903461 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:35:03 [DEBUG   ] Got command RPT from MCS: ref #563590
2018-11-19 23:35:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:35:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563590  12 58441 84903665 A  ERRORB¤ff'
2018-11-19 23:35:15 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:35:28 [DEBUG   ] Got command BAM from MCS: ref #563592
2018-11-19 23:35:28 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:35:28 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563592  75 58441 84928021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:35:28 [DEBUG   ] Got command BAM from MCS: ref #563591
2018-11-19 23:35:28 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:35:28 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563591  75 58441 84928022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:35:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363425056000000
2018-11-19 23:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:35:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363425056000000
2018-11-19 23:35:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 23:35:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:35:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-19 23:35:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:35:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:35:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:35:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:35:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:35:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:35:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 23:35:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:35:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:35:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:35:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:35:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:35:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:35:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:35:34 [DEBUG   ] Finished updating board flags in 1.016 seconds
2018-11-19 23:35:35 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:35:46 [DEBUG   ] Got command RPT from MCS: ref #563593
2018-11-19 23:35:46 [DEBUG   ] summary = ERROR
2018-11-19 23:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563593  13 58441 84946533 A  ERRORERROR'
2018-11-19 23:35:46 [DEBUG   ] Got command RPT from MCS: ref #563594
2018-11-19 23:35:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563594  71 58441 84946534 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:35:55 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:36:05 [DEBUG   ] Got command BAM from MCS: ref #563611
2018-11-19 23:36:05 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:36:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563611  75 58441 84965025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:36:05 [DEBUG   ] Got command BAM from MCS: ref #563610
2018-11-19 23:36:05 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:36:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563610  75 58441 84965026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:36:15 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-19 23:36:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:36:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363436816000000
2018-11-19 23:36:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:36:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:36:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:36:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:36:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 23:36:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:36:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:36:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:36:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:36:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:36:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:36:35 [DEBUG   ] Finished updating ant stats in 0.830 seconds
2018-11-19 23:36:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:36:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:36:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:36:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:36:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:36:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:36:35 [CRITICAL] DRX time not updating correctly on beam(s) #4
2018-11-19 23:36:35 [DEBUG   ] Finished updating board flags in 2.045 seconds
2018-11-19 23:36:42 [DEBUG   ] Got command BAM from MCS: ref #563613
2018-11-19 23:36:42 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:36:42 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563613  75 58441 85002025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:36:42 [DEBUG   ] Got command BAM from MCS: ref #563612
2018-11-19 23:36:42 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:36:42 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563612  75 58441 85002026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:36:55 [DEBUG   ] Finished updating ant stats in 0.774 seconds
2018-11-19 23:37:15 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:37:19 [DEBUG   ] Got command BAM from MCS: ref #563619
2018-11-19 23:37:19 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:37:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563619  75 58441 85039021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:37:19 [DEBUG   ] Got command BAM from MCS: ref #563618
2018-11-19 23:37:19 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:37:19 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563618  75 58441 85039022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:37:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:37:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363448772000000
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:37:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:37:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363448772000000
2018-11-19 23:37:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:37:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:37:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:37:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:37:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:37:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:37:34 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:37:34 [DEBUG   ] Finished updating board flags in 1.406 seconds
2018-11-19 23:37:35 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:37:55 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:37:56 [DEBUG   ] Got command BAM from MCS: ref #563620
2018-11-19 23:37:56 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:37:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563620  75 58441 85076029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:37:56 [DEBUG   ] Got command BAM from MCS: ref #563621
2018-11-19 23:37:56 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:37:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563621  75 58441 85076030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:38:15 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:38:33 [DEBUG   ] Got command BAM from MCS: ref #563622
2018-11-19 23:38:33 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:38:33 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563622  75 58441 85113029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:38:33 [DEBUG   ] Got command BAM from MCS: ref #563623
2018-11-19 23:38:33 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:38:33 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563623  75 58441 85113030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:38:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:38:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363460532000000
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:38:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:38:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363460532000000
2018-11-19 23:38:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:38:34 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:38:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:38:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:38:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:38:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:38:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:38:34 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:38:34 [DEBUG   ] Finished updating board flags in 1.452 seconds
2018-11-19 23:38:35 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:38:55 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:39:10 [DEBUG   ] Got command BAM from MCS: ref #563624
2018-11-19 23:39:10 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:39:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563624  75 58441 85150029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:39:10 [DEBUG   ] Got command BAM from MCS: ref #563625
2018-11-19 23:39:10 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:39:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563625  75 58441 85150030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:39:15 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:39:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:39:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:39:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363472292000000
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:39:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:39:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363472292000000
2018-11-19 23:39:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:39:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:39:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:39:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:39:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:39:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:39:34 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:39:34 [DEBUG   ] Finished updating board flags in 1.357 seconds
2018-11-19 23:39:35 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:39:47 [DEBUG   ] Got command BAM from MCS: ref #563626
2018-11-19 23:39:47 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:39:47 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563626  75 58441 85187021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:39:47 [DEBUG   ] Got command BAM from MCS: ref #563627
2018-11-19 23:39:47 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:39:47 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563627  75 58441 85187022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:39:55 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:40:02 [DEBUG   ] Got command RPT from MCS: ref #563630
2018-11-19 23:40:02 [DEBUG   ] summary = ERROR
2018-11-19 23:40:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563630  13 58441 85202952 A  ERRORERROR'
2018-11-19 23:40:03 [DEBUG   ] Got command RPT from MCS: ref #563631
2018-11-19 23:40:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 23:40:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563631  71 58441 85203141 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 23:40:03 [DEBUG   ] Got command RPT from MCS: ref #563632
2018-11-19 23:40:03 [DEBUG   ] summary = ERROR
2018-11-19 23:40:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563632  13 58441 85203345 A  ERRORERROR'
2018-11-19 23:40:03 [DEBUG   ] Got command RPT from MCS: ref #563634
2018-11-19 23:40:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 23:40:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563634  71 58441 85203549 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 23:40:03 [DEBUG   ] Got command RPT from MCS: ref #563635
2018-11-19 23:40:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:40:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563635  12 58441 85203753 A  ERRORB¤ff'
2018-11-19 23:40:15 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:40:24 [DEBUG   ] Got command BAM from MCS: ref #563645
2018-11-19 23:40:24 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:40:24 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563645  75 58441 85224029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:40:24 [DEBUG   ] Got command BAM from MCS: ref #563644
2018-11-19 23:40:24 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:40:24 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563644  75 58441 85224030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:40:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:40:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363484052000000
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:40:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:40:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363484052000000
2018-11-19 23:40:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:40:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:40:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:40:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:40:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:40:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:40:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:40:34 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:40:34 [DEBUG   ] Finished updating board flags in 1.260 seconds
2018-11-19 23:40:35 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:40:46 [DEBUG   ] Got command RPT from MCS: ref #563646
2018-11-19 23:40:46 [DEBUG   ] summary = ERROR
2018-11-19 23:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563646  13 58441 85246541 A  ERRORERROR'
2018-11-19 23:40:46 [DEBUG   ] Got command RPT from MCS: ref #563647
2018-11-19 23:40:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-19 23:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563647  71 58441 85246542 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-19 23:40:55 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:41:02 [DEBUG   ] Got command BAM from MCS: ref #563663
2018-11-19 23:41:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:41:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563663  75 58441 85262021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:41:02 [DEBUG   ] Got command BAM from MCS: ref #563664
2018-11-19 23:41:02 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:41:02 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563664  75 58441 85262022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:41:15 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:41:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 23:41:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 23:41:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:41:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:41:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:41:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:41:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:41:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:41:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 23:41:35 [DEBUG   ] Finished updating ant stats in 0.629 seconds
2018-11-19 23:41:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:41:35 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:41:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:41:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:41:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:41:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:41:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:41:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:41:35 [DEBUG   ] Finished updating board flags in 1.788 seconds
2018-11-19 23:41:40 [DEBUG   ] Got command BAM from MCS: ref #563665
2018-11-19 23:41:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:41:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563665  75 58441 85300025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:41:40 [DEBUG   ] Got command BAM from MCS: ref #563666
2018-11-19 23:41:40 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:41:40 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563666  75 58441 85300026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:41:55 [DEBUG   ] Finished updating ant stats in 0.575 seconds
2018-11-19 23:42:15 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:42:18 [DEBUG   ] Got command BAM from MCS: ref #563668
2018-11-19 23:42:18 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:42:18 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563668  75 58441 85338021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:42:18 [DEBUG   ] Got command BAM from MCS: ref #563667
2018-11-19 23:42:18 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:42:18 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563667  75 58441 85338022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:42:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 23:42:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:42:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:42:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:42:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:42:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:42:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:42:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:42:35 [DEBUG   ] Finished updating ant stats in 0.429 seconds
2018-11-19 23:42:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:42:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:42:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:42:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:42:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:42:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:42:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:42:35 [DEBUG   ] Finished updating board flags in 1.614 seconds
2018-11-19 23:42:55 [DEBUG   ] Finished updating ant stats in 0.374 seconds
2018-11-19 23:42:56 [DEBUG   ] Got command BAM from MCS: ref #563670
2018-11-19 23:42:56 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:42:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563670  75 58441 85376025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:42:56 [DEBUG   ] Got command BAM from MCS: ref #563669
2018-11-19 23:42:56 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:42:56 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563669  75 58441 85376026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:43:16 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:43:34 [DEBUG   ] Got command BAM from MCS: ref #563675
2018-11-19 23:43:34 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:43:34 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563675  75 58441 85414033 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:43:34 [DEBUG   ] Got command BAM from MCS: ref #563676
2018-11-19 23:43:34 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:43:34 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563676  75 58441 85414034 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:43:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 23:43:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 23:43:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 23:43:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:43:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363519332000000
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:43:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363519332000000
2018-11-19 23:43:34 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:43:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:43:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:43:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:43:34 [WARNING ] (13, 1, 'bot')
2018-11-19 23:43:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:43:34 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:43:34 [DEBUG   ] Finished updating board flags in 0.923 seconds
2018-11-19 23:43:36 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:43:56 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:44:12 [DEBUG   ] Got command BAM from MCS: ref #563678
2018-11-19 23:44:12 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:44:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563678  75 58441 85452029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:44:12 [DEBUG   ] Got command BAM from MCS: ref #563677
2018-11-19 23:44:12 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:44:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563677  75 58441 85452030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:44:16 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:44:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363531092000000
2018-11-19 23:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:44:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363531092000000
2018-11-19 23:44:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-19 23:44:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:44:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:44:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:44:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:44:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:44:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:44:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:44:35 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:44:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:44:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:44:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:44:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:44:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:44:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:44:35 [DEBUG   ] Finished updating board flags in 1.405 seconds
2018-11-19 23:44:36 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:44:50 [DEBUG   ] Got command BAM from MCS: ref #563679
2018-11-19 23:44:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:44:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563679  75 58441 85490029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:44:50 [DEBUG   ] Got command BAM from MCS: ref #563680
2018-11-19 23:44:50 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:44:50 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563680  75 58441 85490030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:44:56 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:45:03 [DEBUG   ] Got command RPT from MCS: ref #563681
2018-11-19 23:45:03 [DEBUG   ] summary = ERROR
2018-11-19 23:45:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563681  13 58441 85503029 A  ERRORERROR'
2018-11-19 23:45:03 [DEBUG   ] Got command RPT from MCS: ref #563682
2018-11-19 23:45:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:45:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563682  71 58441 85503233 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:45:03 [DEBUG   ] Got command RPT from MCS: ref #563683
2018-11-19 23:45:03 [DEBUG   ] summary = ERROR
2018-11-19 23:45:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563683  13 58441 85503425 A  ERRORERROR'
2018-11-19 23:45:03 [DEBUG   ] Got command RPT from MCS: ref #563684
2018-11-19 23:45:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:45:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563684  71 58441 85503629 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:45:03 [DEBUG   ] Got command RPT from MCS: ref #563685
2018-11-19 23:45:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:45:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563685  12 58441 85503833 A  ERRORB¤ff'
2018-11-19 23:45:16 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:45:28 [DEBUG   ] Got command BAM from MCS: ref #563686
2018-11-19 23:45:28 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:45:28 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563686  75 58441 85528029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:45:28 [DEBUG   ] Got command BAM from MCS: ref #563687
2018-11-19 23:45:28 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:45:28 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563687  75 58441 85528030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:45:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363542852000000
2018-11-19 23:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:45:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363542852000000
2018-11-19 23:45:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:45:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:45:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:45:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:45:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:45:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:45:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:45:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:45:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:45:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:45:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:45:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:45:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:45:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:45:35 [DEBUG   ] Finished updating board flags in 1.381 seconds
2018-11-19 23:45:36 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:45:46 [DEBUG   ] Got command RPT from MCS: ref #563688
2018-11-19 23:45:46 [DEBUG   ] summary = ERROR
2018-11-19 23:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563688  13 58441 85546541 A  ERRORERROR'
2018-11-19 23:45:46 [DEBUG   ] Got command RPT from MCS: ref #563689
2018-11-19 23:45:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563689  71 58441 85546542 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:45:56 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:46:06 [DEBUG   ] Got command BAM from MCS: ref #563705
2018-11-19 23:46:06 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:46:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563705  75 58441 85566021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:46:06 [DEBUG   ] Got command BAM from MCS: ref #563706
2018-11-19 23:46:06 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:46:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563706  75 58441 85566022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:46:16 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:46:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363554612000000
2018-11-19 23:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:46:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363554612000000
2018-11-19 23:46:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:46:34 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:46:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:46:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:46:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:46:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:46:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:46:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:46:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:46:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:46:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:46:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:46:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:46:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:46:35 [DEBUG   ] Finished updating board flags in 1.216 seconds
2018-11-19 23:46:36 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:46:44 [DEBUG   ] Got command BAM from MCS: ref #563711
2018-11-19 23:46:44 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:46:44 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563711  75 58441 85604025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:46:44 [DEBUG   ] Got command BAM from MCS: ref #563712
2018-11-19 23:46:44 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:46:44 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563712  75 58441 85604026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:46:56 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:47:16 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:47:22 [DEBUG   ] Got command BAM from MCS: ref #563714
2018-11-19 23:47:22 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:47:22 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563714  75 58441 85642029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:47:22 [DEBUG   ] Got command BAM from MCS: ref #563713
2018-11-19 23:47:22 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:47:22 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563713  75 58441 85642030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:47:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363566372000000
2018-11-19 23:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:47:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363566372000000
2018-11-19 23:47:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 23:47:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-19 23:47:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:47:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:47:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:47:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:47:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:47:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:47:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:47:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:47:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:47:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:47:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:47:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:47:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:47:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:47:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:47:35 [DEBUG   ] Finished updating board flags in 1.189 seconds
2018-11-19 23:47:36 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:47:56 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:48:00 [DEBUG   ] Got command BAM from MCS: ref #563715
2018-11-19 23:48:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:48:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563715  75 58441 85680025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:48:00 [DEBUG   ] Got command BAM from MCS: ref #563716
2018-11-19 23:48:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:48:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563716  75 58441 85680026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:48:16 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:48:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:48:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:48:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:48:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:48:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:48:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:48:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:48:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:48:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:48:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:48:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:48:36 [DEBUG   ] Finished updating ant stats in 1.236 seconds
2018-11-19 23:48:36 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:48:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:48:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:48:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:48:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:48:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:48:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:48:36 [DEBUG   ] Finished updating board flags in 2.118 seconds
2018-11-19 23:48:38 [DEBUG   ] Got command BAM from MCS: ref #563718
2018-11-19 23:48:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:48:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563718  75 58441 85718021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:48:38 [DEBUG   ] Got command BAM from MCS: ref #563717
2018-11-19 23:48:38 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:48:38 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563717  75 58441 85718022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:48:56 [DEBUG   ] Finished updating ant stats in 1.167 seconds
2018-11-19 23:49:16 [DEBUG   ] Got command BAM from MCS: ref #563724
2018-11-19 23:49:16 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:49:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563724  75 58441 85756022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:49:16 [DEBUG   ] Got command BAM from MCS: ref #563723
2018-11-19 23:49:16 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:49:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563723  75 58441 85756022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:49:16 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 23:49:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:49:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:49:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:49:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:49:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:49:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 23:49:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:49:35 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:49:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:49:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:49:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:49:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:49:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:49:36 [DEBUG   ] Finished updating ant stats in 1.026 seconds
2018-11-19 23:49:36 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:49:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:49:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:49:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:49:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:49:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:49:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:49:36 [DEBUG   ] Finished updating board flags in 1.920 seconds
2018-11-19 23:49:54 [DEBUG   ] Got command BAM from MCS: ref #563727
2018-11-19 23:49:54 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:49:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563727  75 58441 85794025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:49:54 [DEBUG   ] Got command BAM from MCS: ref #563728
2018-11-19 23:49:54 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:49:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563728  75 58441 85794026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:49:56 [DEBUG   ] Finished updating ant stats in 0.974 seconds
2018-11-19 23:50:03 [DEBUG   ] Got command RPT from MCS: ref #563729
2018-11-19 23:50:03 [DEBUG   ] summary = ERROR
2018-11-19 23:50:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563729  13 58441 85803110 A  ERRORERROR'
2018-11-19 23:50:03 [DEBUG   ] Got command RPT from MCS: ref #563730
2018-11-19 23:50:03 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 23:50:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563730  98 58441 85803314 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 23:50:03 [DEBUG   ] Got command RPT from MCS: ref #563731
2018-11-19 23:50:03 [DEBUG   ] summary = ERROR
2018-11-19 23:50:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563731  13 58441 85803518 A  ERRORERROR'
2018-11-19 23:50:03 [DEBUG   ] Got command RPT from MCS: ref #563733
2018-11-19 23:50:03 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 23:50:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563733  98 58441 85803721 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 23:50:03 [DEBUG   ] Got command RPT from MCS: ref #563735
2018-11-19 23:50:03 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:50:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563735  12 58441 85803914 A  ERRORB¤ff'
2018-11-19 23:50:16 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-19 23:50:32 [DEBUG   ] Got command BAM from MCS: ref #563739
2018-11-19 23:50:32 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:50:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563739  75 58441 85832022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:50:32 [DEBUG   ] Got command BAM from MCS: ref #563740
2018-11-19 23:50:32 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:50:32 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563740  75 58441 85832022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:50:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:50:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:50:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:50:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:50:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:50:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 23:50:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:50:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:50:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:50:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:50:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:50:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:50:36 [DEBUG   ] Finished updating ant stats in 0.826 seconds
2018-11-19 23:50:36 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:50:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 23:50:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:50:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:50:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:50:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:50:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:50:36 [DEBUG   ] Finished updating board flags in 1.806 seconds
2018-11-19 23:50:46 [DEBUG   ] Got command RPT from MCS: ref #563741
2018-11-19 23:50:46 [DEBUG   ] summary = ERROR
2018-11-19 23:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563741  13 58441 85846541 A  ERRORERROR'
2018-11-19 23:50:46 [DEBUG   ] Got command RPT from MCS: ref #563742
2018-11-19 23:50:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-19 23:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563742  98 58441 85846542 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-19 23:50:56 [DEBUG   ] Finished updating ant stats in 0.774 seconds
2018-11-19 23:51:10 [DEBUG   ] Got command BAM from MCS: ref #563758
2018-11-19 23:51:10 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:51:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563758  75 58441 85870025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:51:10 [DEBUG   ] Got command BAM from MCS: ref #563759
2018-11-19 23:51:10 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:51:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563759  75 58441 85870026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:51:16 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:51:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:51:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:51:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:51:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363613608000000
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:51:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:51:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363613608000000
2018-11-19 23:51:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-19 23:51:35 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:51:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:51:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:51:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:51:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:51:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:51:35 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:51:35 [DEBUG   ] Finished updating board flags in 1.234 seconds
2018-11-19 23:51:36 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:51:48 [DEBUG   ] Got command BAM from MCS: ref #563761
2018-11-19 23:51:48 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:51:48 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563761  75 58441 85908025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:51:48 [DEBUG   ] Got command BAM from MCS: ref #563760
2018-11-19 23:51:48 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:51:48 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563760  75 58441 85908026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:51:56 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:52:16 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-19 23:52:26 [DEBUG   ] Got command BAM from MCS: ref #563766
2018-11-19 23:52:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:52:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563766  75 58441 85946029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:52:26 [DEBUG   ] Got command BAM from MCS: ref #563767
2018-11-19 23:52:26 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:52:26 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563767  75 58441 85946030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:52:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:52:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:52:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:52:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:52:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:52:35 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 23:52:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:52:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:52:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:52:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:52:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:52:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:52:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:52:36 [DEBUG   ] Finished updating ant stats in 0.629 seconds
2018-11-19 23:52:36 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:52:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:52:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:52:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:52:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:52:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:52:36 [DEBUG   ] Finished updating board flags in 1.762 seconds
2018-11-19 23:52:56 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-19 23:53:04 [DEBUG   ] Got command BAM from MCS: ref #563768
2018-11-19 23:53:04 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:53:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563768  75 58441 85984029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:53:04 [DEBUG   ] Got command BAM from MCS: ref #563769
2018-11-19 23:53:04 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:53:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563769  75 58441 85984030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:53:16 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:53:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-19 23:53:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 23:53:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:53:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363637128000000
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:53:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:53:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363637128000000
2018-11-19 23:53:35 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 23:53:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:53:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:53:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:53:35 [WARNING ] (13, 1, 'bot')
2018-11-19 23:53:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:53:35 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-19 23:53:35 [DEBUG   ] Finished updating board flags in 0.983 seconds
2018-11-19 23:53:36 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:53:42 [DEBUG   ] Got command BAM from MCS: ref #563770
2018-11-19 23:53:42 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:53:42 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563770  75 58441 86022021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:53:42 [DEBUG   ] Got command BAM from MCS: ref #563771
2018-11-19 23:53:42 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:53:42 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563771  75 58441 86022022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:53:56 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:54:16 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-19 23:54:21 [DEBUG   ] Got command BAM from MCS: ref #563772
2018-11-19 23:54:21 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:54:21 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563772  75 58441 86061021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:54:21 [DEBUG   ] Got command BAM from MCS: ref #563773
2018-11-19 23:54:21 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:54:21 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563773  75 58441 86061022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:54:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:54:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363648888000000
2018-11-19 23:54:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:54:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:54:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:54:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:54:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363648888000000
2018-11-19 23:54:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-19 23:54:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-19 23:54:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-19 23:54:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:54:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:54:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:54:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:54:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:54:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:54:36 [DEBUG   ] Finished updating ant stats in 0.429 seconds
2018-11-19 23:54:36 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:54:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:54:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:54:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:54:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:54:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:54:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:54:36 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:54:36 [DEBUG   ] Finished updating board flags in 1.439 seconds
2018-11-19 23:54:56 [DEBUG   ] Finished updating ant stats in 0.374 seconds
2018-11-19 23:55:00 [DEBUG   ] Got command BAM from MCS: ref #563774
2018-11-19 23:55:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:55:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563774  75 58441 86100029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:55:00 [DEBUG   ] Got command BAM from MCS: ref #563775
2018-11-19 23:55:00 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:55:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563775  75 58441 86100030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:55:03 [DEBUG   ] Got command RPT from MCS: ref #563776
2018-11-19 23:55:03 [DEBUG   ] summary = ERROR
2018-11-19 23:55:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563776  13 58441 86103201 A  ERRORERROR'
2018-11-19 23:55:03 [DEBUG   ] Got command RPT from MCS: ref #563777
2018-11-19 23:55:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:55:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563777  71 58441 86103393 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:55:03 [DEBUG   ] Got command RPT from MCS: ref #563778
2018-11-19 23:55:03 [DEBUG   ] summary = ERROR
2018-11-19 23:55:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563778  13 58441 86103601 A  ERRORERROR'
2018-11-19 23:55:03 [DEBUG   ] Got command RPT from MCS: ref #563779
2018-11-19 23:55:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:55:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563779  71 58441 86103801 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:55:04 [DEBUG   ] Got command RPT from MCS: ref #563780
2018-11-19 23:55:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-19 23:55:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563780  12 58441 86104005 A  ERRORB¤ff'
2018-11-19 23:55:17 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:55:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:55:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363660648000000
2018-11-19 23:55:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:55:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:55:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:55:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:55:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363660648000000
2018-11-19 23:55:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:55:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:55:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:55:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:55:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:55:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:55:36 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-19 23:55:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:55:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:55:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:55:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:55:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:55:36 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:55:36 [DEBUG   ] Finished updating board flags in 1.349 seconds
2018-11-19 23:55:37 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:55:39 [DEBUG   ] Got command BAM from MCS: ref #563786
2018-11-19 23:55:39 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:55:39 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563786  75 58441 86139025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:55:39 [DEBUG   ] Got command BAM from MCS: ref #563785
2018-11-19 23:55:39 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:55:39 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563785  75 58441 86139026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:55:46 [DEBUG   ] Got command RPT from MCS: ref #563787
2018-11-19 23:55:46 [DEBUG   ] summary = ERROR
2018-11-19 23:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563787  13 58441 86146537 A  ERRORERROR'
2018-11-19 23:55:46 [DEBUG   ] Got command RPT from MCS: ref #563788
2018-11-19 23:55:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-19 23:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563788  71 58441 86146538 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-19 23:55:57 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:56:17 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:56:18 [DEBUG   ] Got command BAM from MCS: ref #563804
2018-11-19 23:56:18 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:56:18 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563804  75 58441 86178025 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:56:18 [DEBUG   ] Got command BAM from MCS: ref #563805
2018-11-19 23:56:18 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:56:18 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563805  75 58441 86178026 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:56:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363672408000000
2018-11-19 23:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:56:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363672408000000
2018-11-19 23:56:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:56:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:56:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:56:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:56:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:56:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:56:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 23:56:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-19 23:56:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:56:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:56:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:56:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:56:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:56:36 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:56:36 [DEBUG   ] Finished updating board flags in 1.218 seconds
2018-11-19 23:56:37 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:56:57 [DEBUG   ] Got command BAM from MCS: ref #563807
2018-11-19 23:56:57 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:56:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563807  75 58441 86217030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:56:57 [DEBUG   ] Got command BAM from MCS: ref #563806
2018-11-19 23:56:57 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:56:57 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563806  75 58441 86217030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:56:57 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-19 23:57:17 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:57:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363684168000000
2018-11-19 23:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:57:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363684168000000
2018-11-19 23:57:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:57:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:57:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:57:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:57:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:57:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:57:36 [DEBUG   ] Got command BAM from MCS: ref #563808
2018-11-19 23:57:36 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:57:36 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563808  75 58441 86256029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:57:36 [DEBUG   ] Got command BAM from MCS: ref #563809
2018-11-19 23:57:36 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:57:36 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563809  75 58441 86256030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:57:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 23:57:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:57:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:57:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:57:36 [WARNING ] (13, 1, 'bot')
2018-11-19 23:57:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:57:36 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-19 23:57:36 [DEBUG   ] Finished updating board flags in 1.185 seconds
2018-11-19 23:57:37 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:57:57 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:58:15 [DEBUG   ] Got command BAM from MCS: ref #563815
2018-11-19 23:58:15 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:58:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563815  75 58441 86295029 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:58:15 [DEBUG   ] Got command BAM from MCS: ref #563814
2018-11-19 23:58:15 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:58:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563814  75 58441 86295030 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:58:17 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-19 23:58:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:58:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:58:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:58:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:58:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:58:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:58:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-19 23:58:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:58:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:58:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:58:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:58:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:58:37 [DEBUG   ] Finished updating ant stats in 1.230 seconds
2018-11-19 23:58:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-19 23:58:37 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 23:58:37 [WARNING ] (13, 1, 'bot')
2018-11-19 23:58:37 [WARNING ] (13, 1, 'bot')
2018-11-19 23:58:37 [WARNING ] (13, 1, 'bot')
2018-11-19 23:58:37 [WARNING ] (13, 1, 'bot')
2018-11-19 23:58:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:58:37 [DEBUG   ] Finished updating board flags in 2.042 seconds
2018-11-19 23:58:54 [DEBUG   ] Got command BAM from MCS: ref #563817
2018-11-19 23:58:54 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:58:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563817  75 58441 86334021 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:58:54 [DEBUG   ] Got command BAM from MCS: ref #563816
2018-11-19 23:58:54 [DEBUG   ] bam = exited with status False, code 0x0d
2018-11-19 23:58:54 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_BAM   563816  75 58441 86334022 R  ERROR0x0D! BAM: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:58:57 [DEBUG   ] Finished updating ant stats in 1.174 seconds
2018-11-19 23:59:10 [DEBUG   ] Got command STP from MCS: ref #563818
2018-11-19 23:59:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   563818  75 58441 86350121 R  ERROR0x0D! STP: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:59:10 [DEBUG   ] Got command STP from MCS: ref #563819
2018-11-19 23:59:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_STP   563819  75 58441 86350122 R  ERROR0x0D! STP: Beamformer sub-subsystem for requested beam is not ready'
2018-11-19 23:59:17 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-19 23:59:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-19 23:59:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-19 23:59:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-19 23:59:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-19 23:59:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-19 23:59:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-19 23:59:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-19 23:59:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-19 23:59:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-19 23:59:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-19 23:59:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-19 23:59:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-19 23:59:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-19 23:59:37 [DEBUG   ] Finished updating ant stats in 1.029 seconds
2018-11-19 23:59:37 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-19 23:59:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-19 23:59:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-19 23:59:37 [WARNING ] (13, 1, 'bot')
2018-11-19 23:59:37 [WARNING ] (13, 1, 'bot')
2018-11-19 23:59:37 [WARNING ] (13, 1, 'bot')
2018-11-19 23:59:37 [WARNING ] (13, 1, 'bot')
2018-11-19 23:59:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-19 23:59:37 [DEBUG   ] Finished updating board flags in 2.007 seconds
2018-11-19 23:59:57 [DEBUG   ] Finished updating ant stats in 0.974 seconds
2018-11-20 00:00:03 [DEBUG   ] Got command RPT from MCS: ref #563822
2018-11-20 00:00:03 [DEBUG   ] summary = ERROR
2018-11-20 00:00:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563822  13 58442     3277 A  ERRORERROR'
2018-11-20 00:00:03 [DEBUG   ] Got command RPT from MCS: ref #563823
2018-11-20 00:00:03 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 00:00:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563823  98 58442     3485 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 00:00:03 [DEBUG   ] Got command RPT from MCS: ref #563824
2018-11-20 00:00:03 [DEBUG   ] summary = ERROR
2018-11-20 00:00:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563824  13 58442     3685 A  ERRORERROR'
2018-11-20 00:00:03 [DEBUG   ] Got command RPT from MCS: ref #563826
2018-11-20 00:00:03 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 00:00:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563826  98 58442     3889 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 00:00:04 [DEBUG   ] Got command RPT from MCS: ref #563828
2018-11-20 00:00:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:00:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563828  12 58442     4081 A  ERRORB£ff'
2018-11-20 00:00:17 [DEBUG   ] Finished updating ant stats in 0.795 seconds
2018-11-20 00:00:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:00:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:00:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:00:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:00:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:00:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:00:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:00:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:00:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:00:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:00:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:00:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-20 00:00:37 [DEBUG   ] Finished updating ant stats in 0.628 seconds
2018-11-20 00:00:37 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:00:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:00:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:00:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:00:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:00:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:00:37 [DEBUG   ] Finished updating board flags in 1.983 seconds
2018-11-20 00:00:46 [DEBUG   ] Got command RPT from MCS: ref #563832
2018-11-20 00:00:46 [DEBUG   ] summary = ERROR
2018-11-20 00:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563832  13 58442    46541 A  ERRORERROR'
2018-11-20 00:00:46 [DEBUG   ] Got command RPT from MCS: ref #563833
2018-11-20 00:00:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 00:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563833  98 58442    46542 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 00:00:57 [DEBUG   ] Finished updating ant stats in 0.574 seconds
2018-11-20 00:01:17 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:01:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:01:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363731404000000
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:01:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:01:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363731404000000
2018-11-20 00:01:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:01:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:01:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:01:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:01:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:01:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:01:36 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:01:36 [DEBUG   ] Finished updating board flags in 1.254 seconds
2018-11-20 00:01:37 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-20 00:01:57 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:02:17 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:02:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:02:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363743164000000
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:02:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:02:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363743164000000
2018-11-20 00:02:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:02:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:02:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:02:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:02:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:02:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:02:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:02:36 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:02:36 [DEBUG   ] Finished updating board flags in 1.235 seconds
2018-11-20 00:02:37 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:02:57 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:03:17 [DEBUG   ] Finished updating ant stats in 0.394 seconds
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:03:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:03:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363754924000000
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:03:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:03:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363754924000000
2018-11-20 00:03:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:03:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:03:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:03:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:03:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:03:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:03:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:03:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:03:36 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:03:36 [DEBUG   ] Finished updating board flags in 1.058 seconds
2018-11-20 00:03:38 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:03:58 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 00:04:18 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:04:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:04:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363766684000000
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:04:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:04:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363766684000000
2018-11-20 00:04:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:04:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:04:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:04:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:04:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:04:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:04:36 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:04:36 [DEBUG   ] Finished updating board flags in 1.039 seconds
2018-11-20 00:04:38 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 00:04:58 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:05:03 [DEBUG   ] Got command RPT from MCS: ref #563857
2018-11-20 00:05:03 [DEBUG   ] summary = ERROR
2018-11-20 00:05:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563857  13 58442   303369 A  ERRORERROR'
2018-11-20 00:05:03 [DEBUG   ] Got command RPT from MCS: ref #563858
2018-11-20 00:05:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:05:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563858  71 58442   303561 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:05:03 [DEBUG   ] Got command RPT from MCS: ref #563859
2018-11-20 00:05:03 [DEBUG   ] summary = ERROR
2018-11-20 00:05:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563859  13 58442   303765 A  ERRORERROR'
2018-11-20 00:05:03 [DEBUG   ] Got command RPT from MCS: ref #563860
2018-11-20 00:05:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:05:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563860  71 58442   303969 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:05:04 [DEBUG   ] Got command RPT from MCS: ref #563861
2018-11-20 00:05:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:05:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563861  12 58442   304173 A  ERRORB£ff'
2018-11-20 00:05:18 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:05:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:05:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363778444000000
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:05:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:05:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363778444000000
2018-11-20 00:05:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:05:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:05:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:05:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:05:36 [WARNING ] (13, 1, 'bot')
2018-11-20 00:05:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:05:36 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:05:36 [DEBUG   ] Finished updating board flags in 0.983 seconds
2018-11-20 00:05:38 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:05:46 [DEBUG   ] Got command RPT from MCS: ref #563862
2018-11-20 00:05:46 [DEBUG   ] summary = ERROR
2018-11-20 00:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563862  13 58442   346545 A  ERRORERROR'
2018-11-20 00:05:46 [DEBUG   ] Got command RPT from MCS: ref #563863
2018-11-20 00:05:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563863  71 58442   346546 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:05:58 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:06:11 [DEBUG   ] Got command TBW from MCS: ref #563880
2018-11-20 00:06:11 [DEBUG   ] tbw = exited with status True, code 0x00
2018-11-20 00:06:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_TBW   563880   8 58442   371269 A  ERROR'
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa16 command /usr/local/bin/lwa16 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa27 command /usr/local/bin/lwa27 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa24 command /usr/local/bin/lwa24 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa23 command /usr/local/bin/lwa23 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa18 command /usr/local/bin/lwa18 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa10 command /usr/local/bin/lwa10 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa32 command /usr/local/bin/lwa32 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa11 command /usr/local/bin/lwa11 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa30 command /usr/local/bin/lwa30 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa12 command /usr/local/bin/lwa12 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa8 command /usr/local/bin/lwa8 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa13 command /usr/local/bin/lwa13 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa26 command /usr/local/bin/lwa26 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa14 command /usr/local/bin/lwa14 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa34 command /usr/local/bin/lwa34 pkill tbn
2018-11-20 00:06:12 [DEBUG   ] sendHostnameCommand - Sent hostname lwa22 command /usr/local/bin/lwa22 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa20 command /usr/local/bin/lwa20 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa1 command /usr/local/bin/lwa1 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa5 command /usr/local/bin/lwa5 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa25 command /usr/local/bin/lwa25 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa35 command /usr/local/bin/lwa35 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa19 command /usr/local/bin/lwa19 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa21 command /usr/local/bin/lwa21 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa6 command /usr/local/bin/lwa6 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa7 command /usr/local/bin/lwa7 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa33 command /usr/local/bin/lwa33 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa15 command /usr/local/bin/lwa15 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] sendHostnameCommand - Sent hostname lwa28 command /usr/local/bin/lwa28 pkill tbn
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa27', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '1', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa24', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '2', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa23', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '3', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa18', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '4', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa10', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '5', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa32', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '6', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa11', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '7', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa30', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '8', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa12', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '9', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa8', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '10', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa13', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '11', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa26', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '12', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa14', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '13', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa22', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '14', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa20', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '15', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa1', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '16', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa5', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '17', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa25', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '18', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa35', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '19', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa19', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '20', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa21', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '21', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa6', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '22', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa7', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '23', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa33', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '24', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa15', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '25', '12000000']
2018-11-20 00:06:13 [DEBUG   ] spawn_process: at 2018-11-20 00:06:13 running ['/usr/local/bin/lwa28', '/lwa/software/tbw', '192.168.0.37', '10000', '12', '0', '26', '12000000']
2018-11-20 00:06:13 [INFO    ] Finished spawning TBW
2018-11-20 00:06:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:06:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363790204000000
2018-11-20 00:06:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:06:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:06:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:06:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:06:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363790204000000
2018-11-20 00:06:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:06:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:06:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:06:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:06:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:06:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:06:37 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:06:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:06:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:06:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:06:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:06:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:06:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:06:37 [DEBUG   ] Finished updating board flags in 1.460 seconds
2018-11-20 00:07:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:07:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363801964000000
2018-11-20 00:07:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:07:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:07:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:07:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:07:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363801964000000
2018-11-20 00:07:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 00:07:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:07:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:07:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:07:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:07:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:07:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:07:37 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:07:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:07:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:07:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:07:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:07:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:07:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:07:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:07:37 [DEBUG   ] Finished updating board flags in 1.478 seconds
2018-11-20 00:08:01 [INFO    ] __waitForTBW - TBW finished in 108.436 s
2018-11-20 00:08:02 [DEBUG   ] Finished updating ant stats in 0.320 seconds
2018-11-20 00:08:02 [INFO    ] Started antennas statistics background thread
2018-11-20 00:08:23 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 00:08:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:08:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363813724000000
2018-11-20 00:08:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:08:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:08:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:08:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:08:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363813724000000
2018-11-20 00:08:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:08:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:08:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:08:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:08:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:08:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:08:37 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:08:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-20 00:08:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:08:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:08:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:08:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:08:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:08:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:08:37 [DEBUG   ] Finished updating board flags in 1.227 seconds
2018-11-20 00:08:43 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:09:03 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:09:23 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:09:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:09:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363825484000000
2018-11-20 00:09:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:09:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:09:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:09:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:09:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363825484000000
2018-11-20 00:09:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:09:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:09:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:09:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:09:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:09:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:09:37 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:09:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:09:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:09:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:09:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:09:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:09:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:09:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:09:37 [DEBUG   ] Finished updating board flags in 1.196 seconds
2018-11-20 00:09:43 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 00:10:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:10:03 [DEBUG   ] Got command RPT from MCS: ref #563894
2018-11-20 00:10:03 [DEBUG   ] summary = ERROR
2018-11-20 00:10:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563894  13 58442   603441 A  ERRORERROR'
2018-11-20 00:10:03 [DEBUG   ] Got command RPT from MCS: ref #563895
2018-11-20 00:10:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:10:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563895  71 58442   603645 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:10:03 [DEBUG   ] Got command RPT from MCS: ref #563896
2018-11-20 00:10:03 [DEBUG   ] summary = ERROR
2018-11-20 00:10:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563896  13 58442   603849 A  ERRORERROR'
2018-11-20 00:10:04 [DEBUG   ] Got command RPT from MCS: ref #563898
2018-11-20 00:10:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:10:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563898  71 58442   604041 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:10:04 [DEBUG   ] Got command RPT from MCS: ref #563899
2018-11-20 00:10:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:10:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563899  12 58442   604245 A  ERRORB£ff'
2018-11-20 00:10:23 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:10:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363837244000000
2018-11-20 00:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:10:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363837244000000
2018-11-20 00:10:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 00:10:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:10:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:10:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:10:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:10:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:10:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:10:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:10:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 00:10:37 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:10:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:10:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:10:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:10:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:10:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:10:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:10:37 [DEBUG   ] Finished updating board flags in 1.012 seconds
2018-11-20 00:10:43 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:10:46 [DEBUG   ] Got command RPT from MCS: ref #563908
2018-11-20 00:10:46 [DEBUG   ] summary = ERROR
2018-11-20 00:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563908  13 58442   646549 A  ERRORERROR'
2018-11-20 00:10:46 [DEBUG   ] Got command RPT from MCS: ref #563909
2018-11-20 00:10:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563909  71 58442   646550 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:11:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:11:23 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:11:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363849004000000
2018-11-20 00:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:11:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363849004000000
2018-11-20 00:11:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 00:11:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:11:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:11:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:11:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:11:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:11:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:11:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:11:37 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:11:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:11:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:11:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:11:37 [WARNING ] (13, 1, 'bot')
2018-11-20 00:11:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:11:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:11:37 [DEBUG   ] Finished updating board flags in 0.989 seconds
2018-11-20 00:11:43 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 00:12:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:12:23 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:12:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:12:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:12:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:12:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:12:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:12:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:12:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:12:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:12:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:12:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:12:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:12:38 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:12:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:12:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:12:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:12:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:12:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:12:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:12:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:12:38 [DEBUG   ] Finished updating board flags in 1.829 seconds
2018-11-20 00:12:43 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:13:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:13:23 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 00:13:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:13:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:13:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:13:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:13:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:13:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 00:13:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:13:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:13:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:13:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:13:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:13:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:13:38 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:13:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:13:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:13:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:13:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:13:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:13:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:13:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:13:38 [DEBUG   ] Finished updating board flags in 1.787 seconds
2018-11-20 00:13:43 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 00:14:03 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 00:14:23 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 00:14:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:14:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:14:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:14:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:14:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:14:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:14:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:14:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:14:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:14:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:14:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:14:38 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:14:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:14:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:14:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:14:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:14:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:14:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:14:38 [DEBUG   ] Finished updating board flags in 1.591 seconds
2018-11-20 00:14:43 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:15:03 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:15:03 [DEBUG   ] Got command RPT from MCS: ref #563929
2018-11-20 00:15:03 [DEBUG   ] summary = ERROR
2018-11-20 00:15:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563929  13 58442   903521 A  ERRORERROR'
2018-11-20 00:15:03 [DEBUG   ] Got command RPT from MCS: ref #563930
2018-11-20 00:15:03 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 00:15:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563930  98 58442   903725 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 00:15:03 [DEBUG   ] Got command RPT from MCS: ref #563931
2018-11-20 00:15:03 [DEBUG   ] summary = ERROR
2018-11-20 00:15:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563931  13 58442   903929 A  ERRORERROR'
2018-11-20 00:15:04 [DEBUG   ] Got command RPT from MCS: ref #563932
2018-11-20 00:15:04 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 00:15:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563932  98 58442   904133 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 00:15:04 [DEBUG   ] Got command RPT from MCS: ref #563933
2018-11-20 00:15:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:15:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563933  12 58442   904325 A  ERRORB£ff'
2018-11-20 00:15:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:15:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:15:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:15:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:15:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:15:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:15:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363896240000000
2018-11-20 00:15:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 00:15:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:15:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:15:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:15:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:15:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:15:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:15:38 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:15:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:15:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:15:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:15:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:15:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:15:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:15:38 [CRITICAL] DRX time not updating correctly on beam(s) #3
2018-11-20 00:15:38 [DEBUG   ] Finished updating board flags in 1.417 seconds
2018-11-20 00:15:43 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:15:46 [DEBUG   ] Got command RPT from MCS: ref #563934
2018-11-20 00:15:46 [DEBUG   ] summary = ERROR
2018-11-20 00:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563934  13 58442   946541 A  ERRORERROR'
2018-11-20 00:15:46 [DEBUG   ] Got command RPT from MCS: ref #563935
2018-11-20 00:15:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #3
2018-11-20 00:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563935  67 58442   946542 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #3'
2018-11-20 00:16:03 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:16:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:16:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:16:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363908000000000
2018-11-20 00:16:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:16:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:16:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:16:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:16:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363908000000000
2018-11-20 00:16:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 00:16:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 00:16:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-20 00:16:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:16:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:16:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:16:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:16:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:16:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:16:38 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:16:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:16:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:16:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:16:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:16:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:16:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:16:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:16:38 [DEBUG   ] Finished updating board flags in 1.430 seconds
2018-11-20 00:16:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:17:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:17:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:17:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:17:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363919760000000
2018-11-20 00:17:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:17:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:17:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:17:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:17:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363919760000000
2018-11-20 00:17:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:17:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:17:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:17:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:17:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:17:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:17:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:17:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-20 00:17:38 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:17:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:17:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:17:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:17:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:17:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:17:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:17:38 [DEBUG   ] Finished updating board flags in 1.361 seconds
2018-11-20 00:17:43 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:18:03 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:18:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:18:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:18:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363931520000000
2018-11-20 00:18:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:18:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:18:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:18:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:18:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363931520000000
2018-11-20 00:18:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:18:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:18:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:18:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:18:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:18:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:18:38 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:18:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:18:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:18:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:18:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:18:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:18:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:18:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:18:38 [DEBUG   ] Finished updating board flags in 1.240 seconds
2018-11-20 00:18:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:19:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:19:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:19:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:19:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363943280000000
2018-11-20 00:19:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:19:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:19:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:19:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:19:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363943280000000
2018-11-20 00:19:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:19:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:19:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:19:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:19:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:19:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:19:38 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:19:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:19:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:19:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:19:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:19:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:19:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:19:38 [DEBUG   ] Finished updating board flags in 1.219 seconds
2018-11-20 00:19:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:20:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:20:03 [DEBUG   ] Got command RPT from MCS: ref #563972
2018-11-20 00:20:03 [DEBUG   ] summary = ERROR
2018-11-20 00:20:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563972  13 58442  1203609 A  ERRORERROR'
2018-11-20 00:20:03 [DEBUG   ] Got command RPT from MCS: ref #563973
2018-11-20 00:20:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:20:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563973  71 58442  1203801 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:20:04 [DEBUG   ] Got command RPT from MCS: ref #563974
2018-11-20 00:20:04 [DEBUG   ] summary = ERROR
2018-11-20 00:20:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563974  13 58442  1204005 A  ERRORERROR'
2018-11-20 00:20:04 [DEBUG   ] Got command RPT from MCS: ref #563976
2018-11-20 00:20:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:20:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563976  71 58442  1204209 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:20:04 [DEBUG   ] Got command RPT from MCS: ref #563978
2018-11-20 00:20:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:20:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563978  12 58442  1204413 A  ERRORB¥ff'
2018-11-20 00:20:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:20:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:20:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363955040000000
2018-11-20 00:20:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:20:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:20:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:20:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:20:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363955040000000
2018-11-20 00:20:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:20:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:20:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:20:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:20:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:20:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:20:38 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:20:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:20:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:20:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:20:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:20:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:20:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:20:38 [DEBUG   ] Finished updating board flags in 1.134 seconds
2018-11-20 00:20:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:20:46 [DEBUG   ] Got command RPT from MCS: ref #563982
2018-11-20 00:20:46 [DEBUG   ] summary = ERROR
2018-11-20 00:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563982  13 58442  1246549 A  ERRORERROR'
2018-11-20 00:20:46 [DEBUG   ] Got command RPT from MCS: ref #563983
2018-11-20 00:20:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   563983  71 58442  1246550 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:21:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:21:23 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:21:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:21:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302363966800000000
2018-11-20 00:21:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:21:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:21:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:21:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:21:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302363966800000000
2018-11-20 00:21:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:21:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:21:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:21:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:21:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:21:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:21:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:21:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-20 00:21:38 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:21:38 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:21:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:21:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:21:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:21:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:21:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:21:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:21:38 [DEBUG   ] Finished updating board flags in 1.032 seconds
2018-11-20 00:21:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:22:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:22:23 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:22:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:22:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:22:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:22:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:22:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:22:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:22:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:22:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:22:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:22:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:22:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:22:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 00:22:39 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:22:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:22:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:22:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:22:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:22:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:22:39 [DEBUG   ] Finished updating board flags in 2.012 seconds
2018-11-20 00:22:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:23:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:23:23 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:23:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:23:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302363990516000000
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:23:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:23:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302363990516000000
2018-11-20 00:23:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 00:23:38 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:23:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:23:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:23:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:23:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:23:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:23:38 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:23:38 [DEBUG   ] Finished updating board flags in 1.402 seconds
2018-11-20 00:23:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:24:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:24:23 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:24:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 00:24:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 00:24:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:24:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:24:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364002276000000
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:24:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:24:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364002276000000
2018-11-20 00:24:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 00:24:38 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:24:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:24:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:24:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:24:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:24:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:24:38 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:24:38 [DEBUG   ] Finished updating board flags in 1.474 seconds
2018-11-20 00:24:43 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:25:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:25:03 [DEBUG   ] Got command RPT from MCS: ref #564003
2018-11-20 00:25:03 [DEBUG   ] summary = ERROR
2018-11-20 00:25:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564003  13 58442  1503689 A  ERRORERROR'
2018-11-20 00:25:03 [DEBUG   ] Got command RPT from MCS: ref #564004
2018-11-20 00:25:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:25:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564004  71 58442  1503881 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:25:04 [DEBUG   ] Got command RPT from MCS: ref #564005
2018-11-20 00:25:04 [DEBUG   ] summary = ERROR
2018-11-20 00:25:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564005  13 58442  1504085 A  ERRORERROR'
2018-11-20 00:25:04 [DEBUG   ] Got command RPT from MCS: ref #564006
2018-11-20 00:25:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:25:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564006  71 58442  1504289 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:25:04 [DEBUG   ] Got command RPT from MCS: ref #564007
2018-11-20 00:25:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:25:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564007  12 58442  1504493 A  ERRORB¥ff'
2018-11-20 00:25:23 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:25:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 00:25:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 00:25:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 00:25:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:25:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:25:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364014036000000
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:25:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:25:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364014036000000
2018-11-20 00:25:38 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:25:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:25:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:25:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:25:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:25:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:25:38 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:25:38 [DEBUG   ] Finished updating board flags in 1.379 seconds
2018-11-20 00:25:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:25:46 [DEBUG   ] Got command RPT from MCS: ref #564012
2018-11-20 00:25:46 [DEBUG   ] summary = ERROR
2018-11-20 00:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564012  13 58442  1546541 A  ERRORERROR'
2018-11-20 00:25:46 [DEBUG   ] Got command RPT from MCS: ref #564013
2018-11-20 00:25:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564013  71 58442  1546542 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:26:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:26:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:26:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 00:26:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:26:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364025796000000
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:26:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:26:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364025796000000
2018-11-20 00:26:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 00:26:38 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:26:38 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:26:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:26:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:26:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:26:38 [WARNING ] (13, 1, 'bot')
2018-11-20 00:26:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:26:38 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:26:38 [DEBUG   ] Finished updating board flags in 1.146 seconds
2018-11-20 00:26:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:27:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:27:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:27:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:27:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:27:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:27:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:27:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:27:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 00:27:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 00:27:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:27:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:27:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:27:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:27:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:27:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:27:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:27:39 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:27:39 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:27:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:27:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:27:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:27:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:27:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:27:39 [DEBUG   ] Finished updating board flags in 1.585 seconds
2018-11-20 00:27:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:28:03 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:28:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:28:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:28:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364049316000000
2018-11-20 00:28:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:28:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:28:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:28:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:28:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364049316000000
2018-11-20 00:28:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:28:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:28:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:28:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:28:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:28:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:28:39 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:28:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:28:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:28:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:28:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:28:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:28:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:28:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:28:39 [DEBUG   ] Finished updating board flags in 1.429 seconds
2018-11-20 00:28:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:29:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:29:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:29:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:29:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364061076000000
2018-11-20 00:29:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:29:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:29:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:29:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:29:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364061076000000
2018-11-20 00:29:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:29:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 00:29:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:29:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:29:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:29:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:29:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:29:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 00:29:39 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:29:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:29:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:29:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:29:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:29:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:29:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:29:39 [DEBUG   ] Finished updating board flags in 1.355 seconds
2018-11-20 00:29:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:30:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:30:03 [DEBUG   ] Got command RPT from MCS: ref #564035
2018-11-20 00:30:03 [DEBUG   ] summary = ERROR
2018-11-20 00:30:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564035  13 58442  1803769 A  ERRORERROR'
2018-11-20 00:30:03 [DEBUG   ] Got command RPT from MCS: ref #564036
2018-11-20 00:30:03 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:30:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564036  71 58442  1803961 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:30:04 [DEBUG   ] Got command RPT from MCS: ref #564037
2018-11-20 00:30:04 [DEBUG   ] summary = ERROR
2018-11-20 00:30:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564037  13 58442  1804165 A  ERRORERROR'
2018-11-20 00:30:04 [DEBUG   ] Got command RPT from MCS: ref #564039
2018-11-20 00:30:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:30:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564039  71 58442  1804369 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:30:04 [DEBUG   ] Got command RPT from MCS: ref #564040
2018-11-20 00:30:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:30:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564040  12 58442  1804573 A  ERRORB¤ff'
2018-11-20 00:30:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:30:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:30:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364072836000000
2018-11-20 00:30:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:30:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:30:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:30:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:30:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364072836000000
2018-11-20 00:30:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 00:30:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:30:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:30:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:30:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:30:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:30:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:30:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 00:30:39 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:30:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:30:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:30:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:30:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:30:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:30:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:30:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:30:39 [DEBUG   ] Finished updating board flags in 1.240 seconds
2018-11-20 00:30:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:30:46 [DEBUG   ] Got command RPT from MCS: ref #564045
2018-11-20 00:30:46 [DEBUG   ] summary = ERROR
2018-11-20 00:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564045  13 58442  1846545 A  ERRORERROR'
2018-11-20 00:30:46 [DEBUG   ] Got command RPT from MCS: ref #564046
2018-11-20 00:30:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564046  71 58442  1846546 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:31:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:31:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:31:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:31:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364084596000000
2018-11-20 00:31:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:31:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:31:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:31:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:31:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364084596000000
2018-11-20 00:31:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:31:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:31:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:31:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:31:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:31:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:31:39 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:31:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:31:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:31:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:31:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:31:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:31:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:31:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:31:39 [DEBUG   ] Finished updating board flags in 1.167 seconds
2018-11-20 00:31:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:32:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:32:23 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:32:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364096356000000
2018-11-20 00:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:32:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364096356000000
2018-11-20 00:32:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 00:32:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 00:32:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:32:38 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:32:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 00:32:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:32:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:32:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:32:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:32:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:32:39 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:32:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:32:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:32:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:32:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:32:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:32:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:32:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:32:39 [DEBUG   ] Finished updating board flags in 1.022 seconds
2018-11-20 00:32:43 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:33:03 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:33:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 00:33:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:33:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:33:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:33:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:33:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:33:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:33:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:33:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:33:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:33:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:33:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:33:40 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:33:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:33:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:33:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:33:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:33:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:33:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:33:40 [DEBUG   ] Finished updating board flags in 2.024 seconds
2018-11-20 00:33:43 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:34:03 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:34:23 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:34:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:34:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364120072000000
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:34:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:34:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364120072000000
2018-11-20 00:34:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-20 00:34:39 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:34:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:34:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:34:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:34:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:34:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:34:39 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:34:39 [DEBUG   ] Finished updating board flags in 1.392 seconds
2018-11-20 00:34:43 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 00:35:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:35:03 [DEBUG   ] Got command RPT from MCS: ref #564070
2018-11-20 00:35:03 [DEBUG   ] summary = ERROR
2018-11-20 00:35:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564070  13 58442  2103842 A  ERRORERROR'
2018-11-20 00:35:04 [DEBUG   ] Got command RPT from MCS: ref #564071
2018-11-20 00:35:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:35:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564071  71 58442  2104045 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:35:04 [DEBUG   ] Got command RPT from MCS: ref #564072
2018-11-20 00:35:04 [DEBUG   ] summary = ERROR
2018-11-20 00:35:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564072  13 58442  2104249 A  ERRORERROR'
2018-11-20 00:35:04 [DEBUG   ] Got command RPT from MCS: ref #564073
2018-11-20 00:35:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:35:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564073  71 58442  2104442 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:35:04 [DEBUG   ] Got command RPT from MCS: ref #564074
2018-11-20 00:35:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:35:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564074  12 58442  2104645 A  ERRORB£ff'
2018-11-20 00:35:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:35:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 00:35:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:35:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364131832000000
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:35:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:35:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364131832000000
2018-11-20 00:35:39 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:35:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:35:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:35:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:35:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:35:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:35:39 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:35:39 [DEBUG   ] Finished updating board flags in 1.226 seconds
2018-11-20 00:35:43 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 00:35:46 [DEBUG   ] Got command RPT from MCS: ref #564075
2018-11-20 00:35:46 [DEBUG   ] summary = ERROR
2018-11-20 00:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564075  13 58442  2146549 A  ERRORERROR'
2018-11-20 00:35:46 [DEBUG   ] Got command RPT from MCS: ref #564076
2018-11-20 00:35:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564076  71 58442  2146550 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:36:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:36:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:36:39 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 00:36:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:36:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364143592000000
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:36:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:36:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364143592000000
2018-11-20 00:36:39 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:36:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:36:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:36:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:36:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:36:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:36:39 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:36:39 [DEBUG   ] Finished updating board flags in 1.175 seconds
2018-11-20 00:36:43 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 00:37:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:37:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:37:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 00:37:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:37:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364155352000000
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:37:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:37:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364155352000000
2018-11-20 00:37:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 00:37:39 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:37:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:37:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:37:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:37:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:37:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:37:39 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:37:39 [DEBUG   ] Finished updating board flags in 1.036 seconds
2018-11-20 00:37:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:38:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:38:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:38:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:38:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364167112000000
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:38:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:38:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364167112000000
2018-11-20 00:38:39 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:38:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:38:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:38:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:38:39 [WARNING ] (13, 1, 'bot')
2018-11-20 00:38:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:38:39 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:38:39 [DEBUG   ] Finished updating board flags in 0.984 seconds
2018-11-20 00:38:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:39:03 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 00:39:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:39:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:39:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364178872000000
2018-11-20 00:39:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:39:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:39:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:39:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:39:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364178872000000
2018-11-20 00:39:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:39:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:39:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:39:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:39:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:39:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:39:40 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:39:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:39:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:39:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:39:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:39:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:39:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:39:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:39:40 [DEBUG   ] Finished updating board flags in 1.423 seconds
2018-11-20 00:39:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:40:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:40:03 [DEBUG   ] Got command RPT from MCS: ref #564098
2018-11-20 00:40:03 [DEBUG   ] summary = ERROR
2018-11-20 00:40:03 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564098  13 58442  2403925 A  ERRORERROR'
2018-11-20 00:40:04 [DEBUG   ] Got command RPT from MCS: ref #564099
2018-11-20 00:40:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:40:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564099  71 58442  2404129 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:40:04 [DEBUG   ] Got command RPT from MCS: ref #564100
2018-11-20 00:40:04 [DEBUG   ] summary = ERROR
2018-11-20 00:40:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564100  13 58442  2404321 A  ERRORERROR'
2018-11-20 00:40:04 [DEBUG   ] Got command RPT from MCS: ref #564102
2018-11-20 00:40:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:40:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564102  71 58442  2404529 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:40:04 [DEBUG   ] Got command RPT from MCS: ref #564103
2018-11-20 00:40:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:40:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564103  12 58442  2404721 A  ERRORB£ff'
2018-11-20 00:40:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:40:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364190632000000
2018-11-20 00:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:40:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364190632000000
2018-11-20 00:40:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 00:40:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 00:40:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 00:40:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-20 00:40:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:40:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:40:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:40:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:40:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:40:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:40:40 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:40:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:40:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:40:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:40:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:40:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:40:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:40:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:40:40 [DEBUG   ] Finished updating board flags in 1.330 seconds
2018-11-20 00:40:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:40:46 [DEBUG   ] Got command RPT from MCS: ref #564112
2018-11-20 00:40:46 [DEBUG   ] summary = ERROR
2018-11-20 00:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564112  13 58442  2446553 A  ERRORERROR'
2018-11-20 00:40:46 [DEBUG   ] Got command RPT from MCS: ref #564113
2018-11-20 00:40:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564113  71 58442  2446554 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:41:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:41:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:41:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364202392000000
2018-11-20 00:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:41:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364202392000000
2018-11-20 00:41:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:41:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:41:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:41:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:41:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:41:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:41:40 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:41:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:41:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:41:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:41:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:41:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:41:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:41:40 [DEBUG   ] Finished updating board flags in 1.211 seconds
2018-11-20 00:41:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:42:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:42:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:42:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364214152000000
2018-11-20 00:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:42:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364214152000000
2018-11-20 00:42:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-20 00:42:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:42:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-20 00:42:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:42:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:42:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:42:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:42:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:42:40 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:42:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:42:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:42:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:42:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:42:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:42:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:42:40 [DEBUG   ] Finished updating board flags in 1.218 seconds
2018-11-20 00:42:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:43:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:43:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:43:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364225912000000
2018-11-20 00:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:43:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364225912000000
2018-11-20 00:43:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:43:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:43:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:43:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:43:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:43:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:43:40 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:43:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:43:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:43:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:43:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:43:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:43:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:43:40 [DEBUG   ] Finished updating board flags in 1.181 seconds
2018-11-20 00:43:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:44:03 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 00:44:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:44:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364237672000000
2018-11-20 00:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:44:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364237672000000
2018-11-20 00:44:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:44:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:44:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:44:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:44:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:44:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:44:40 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:44:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:44:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:44:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:44:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:44:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:44:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:44:40 [DEBUG   ] Finished updating board flags in 1.044 seconds
2018-11-20 00:44:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:45:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:45:04 [DEBUG   ] Got command RPT from MCS: ref #564133
2018-11-20 00:45:04 [DEBUG   ] summary = ERROR
2018-11-20 00:45:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564133  13 58442  2704005 A  ERRORERROR'
2018-11-20 00:45:04 [DEBUG   ] Got command RPT from MCS: ref #564134
2018-11-20 00:45:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:45:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564134  71 58442  2704209 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:45:04 [DEBUG   ] Got command RPT from MCS: ref #564135
2018-11-20 00:45:04 [DEBUG   ] summary = ERROR
2018-11-20 00:45:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564135  13 58442  2704401 A  ERRORERROR'
2018-11-20 00:45:04 [DEBUG   ] Got command RPT from MCS: ref #564136
2018-11-20 00:45:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:45:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564136  71 58442  2704605 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:45:04 [DEBUG   ] Got command RPT from MCS: ref #564137
2018-11-20 00:45:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:45:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564137  12 58442  2704809 A  ERRORB¥ff'
2018-11-20 00:45:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:45:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:45:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:45:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:45:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:45:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:45:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-20 00:45:40 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 00:45:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 00:45:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:45:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:45:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:45:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:45:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:45:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:45:41 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:45:41 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:45:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:45:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:45:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:45:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:45:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:45:41 [DEBUG   ] Finished updating board flags in 2.035 seconds
2018-11-20 00:45:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:45:46 [DEBUG   ] Got command RPT from MCS: ref #564138
2018-11-20 00:45:46 [DEBUG   ] summary = ERROR
2018-11-20 00:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564138  13 58442  2746545 A  ERRORERROR'
2018-11-20 00:45:46 [DEBUG   ] Got command RPT from MCS: ref #564139
2018-11-20 00:45:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 00:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564139  98 58442  2746546 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 00:46:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:46:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:46:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:46:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:46:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364261388000000
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:46:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:46:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364261388000000
2018-11-20 00:46:40 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:46:40 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:46:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:46:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:46:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:46:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:46:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:46:40 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:46:40 [DEBUG   ] Finished updating board flags in 1.364 seconds
2018-11-20 00:46:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:47:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:47:23 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:47:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 00:47:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 00:47:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:47:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364273148000000
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:47:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:47:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364273148000000
2018-11-20 00:47:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 00:47:40 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:47:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:47:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:47:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:47:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:47:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:47:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:47:40 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:47:40 [DEBUG   ] Finished updating board flags in 1.228 seconds
2018-11-20 00:47:43 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:48:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 00:48:23 [DEBUG   ] Finished updating ant stats in 0.386 seconds
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:48:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 00:48:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:48:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:48:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364284908000000
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:48:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:48:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364284908000000
2018-11-20 00:48:40 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:48:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:48:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:48:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:48:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:48:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:48:40 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:48:40 [DEBUG   ] Finished updating board flags in 1.170 seconds
2018-11-20 00:48:44 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:49:04 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:49:24 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:49:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:49:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:49:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364296668000000
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:49:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:49:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364296668000000
2018-11-20 00:49:40 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:49:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:49:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:49:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:49:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:49:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:49:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:49:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:49:40 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:49:40 [DEBUG   ] Finished updating board flags in 1.031 seconds
2018-11-20 00:49:44 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 00:50:04 [DEBUG   ] Got command RPT from MCS: ref #564165
2018-11-20 00:50:04 [DEBUG   ] summary = ERROR
2018-11-20 00:50:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564165  13 58442  3004085 A  ERRORERROR'
2018-11-20 00:50:04 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 00:50:04 [DEBUG   ] Got command RPT from MCS: ref #564166
2018-11-20 00:50:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:50:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564166  71 58442  3004277 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:50:04 [DEBUG   ] Got command RPT from MCS: ref #564167
2018-11-20 00:50:04 [DEBUG   ] summary = ERROR
2018-11-20 00:50:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564167  13 58442  3004481 A  ERRORERROR'
2018-11-20 00:50:04 [DEBUG   ] Got command RPT from MCS: ref #564169
2018-11-20 00:50:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:50:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564169  71 58442  3004685 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:50:04 [DEBUG   ] Got command RPT from MCS: ref #564170
2018-11-20 00:50:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:50:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564170  12 58442  3004889 A  ERRORB¥ff'
2018-11-20 00:50:24 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:50:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 00:50:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:50:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:50:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364308428000000
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:50:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:50:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364308428000000
2018-11-20 00:50:40 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:50:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:50:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:50:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:50:40 [WARNING ] (13, 1, 'bot')
2018-11-20 00:50:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:50:40 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 00:50:40 [DEBUG   ] Finished updating board flags in 0.994 seconds
2018-11-20 00:50:44 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 00:50:46 [DEBUG   ] Got command RPT from MCS: ref #564175
2018-11-20 00:50:46 [DEBUG   ] summary = ERROR
2018-11-20 00:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564175  13 58442  3046549 A  ERRORERROR'
2018-11-20 00:50:46 [DEBUG   ] Got command RPT from MCS: ref #564176
2018-11-20 00:50:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 00:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564176  71 58442  3046550 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 00:51:04 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:51:24 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 00:51:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:51:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364320188000000
2018-11-20 00:51:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:51:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:51:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:51:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:51:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364320188000000
2018-11-20 00:51:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:51:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:51:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:51:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:51:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:51:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:51:41 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:51:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:51:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:51:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:51:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:51:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:51:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:51:41 [DEBUG   ] Finished updating board flags in 1.417 seconds
2018-11-20 00:51:44 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 00:52:04 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 00:52:24 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 00:52:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:52:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364331948000000
2018-11-20 00:52:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:52:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:52:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:52:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:52:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364331948000000
2018-11-20 00:52:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 00:52:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 00:52:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 00:52:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:52:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 00:52:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:52:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:52:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:52:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:52:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:52:41 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:52:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:52:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:52:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:52:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:52:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:52:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:52:41 [DEBUG   ] Finished updating board flags in 1.412 seconds
2018-11-20 00:52:44 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 00:53:04 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:53:24 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:53:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364343708000000
2018-11-20 00:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:53:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364343708000000
2018-11-20 00:53:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 00:53:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:53:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:53:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:53:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:53:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:53:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:53:41 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:53:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:53:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:53:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:53:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:53:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:53:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:53:41 [DEBUG   ] Finished updating board flags in 1.315 seconds
2018-11-20 00:53:44 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 00:54:04 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:54:24 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 00:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:54:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364355468000000
2018-11-20 00:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:54:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364355468000000
2018-11-20 00:54:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:54:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:54:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:54:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:54:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:54:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:54:41 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:54:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:54:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:54:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:54:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:54:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:54:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:54:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:54:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:54:41 [DEBUG   ] Finished updating board flags in 1.211 seconds
2018-11-20 00:54:44 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 00:55:04 [DEBUG   ] Got command RPT from MCS: ref #564196
2018-11-20 00:55:04 [DEBUG   ] summary = ERROR
2018-11-20 00:55:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564196  13 58442  3304170 A  ERRORERROR'
2018-11-20 00:55:04 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 00:55:04 [DEBUG   ] Got command RPT from MCS: ref #564197
2018-11-20 00:55:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:55:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564197  71 58442  3304361 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:55:04 [DEBUG   ] Got command RPT from MCS: ref #564198
2018-11-20 00:55:04 [DEBUG   ] summary = ERROR
2018-11-20 00:55:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564198  13 58442  3304565 A  ERRORERROR'
2018-11-20 00:55:04 [DEBUG   ] Got command RPT from MCS: ref #564199
2018-11-20 00:55:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:55:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564199  71 58442  3304769 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:55:04 [DEBUG   ] Got command RPT from MCS: ref #564200
2018-11-20 00:55:04 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 00:55:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564200  12 58442  3304961 A  ERRORB¤ff'
2018-11-20 00:55:24 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 00:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:55:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364367228000000
2018-11-20 00:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:55:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364367228000000
2018-11-20 00:55:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:55:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:55:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:55:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:55:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:55:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:55:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:55:41 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:55:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:55:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:55:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:55:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:55:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:55:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:55:41 [DEBUG   ] Finished updating board flags in 1.214 seconds
2018-11-20 00:55:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 00:55:46 [DEBUG   ] Got command RPT from MCS: ref #564205
2018-11-20 00:55:46 [DEBUG   ] summary = ERROR
2018-11-20 00:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564205  13 58442  3346549 A  ERRORERROR'
2018-11-20 00:55:46 [DEBUG   ] Got command RPT from MCS: ref #564206
2018-11-20 00:55:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 00:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564206  71 58442  3346550 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 00:56:04 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 00:56:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 00:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:56:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364378988000000
2018-11-20 00:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:56:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364378988000000
2018-11-20 00:56:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:56:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:56:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:56:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:56:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:56:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:56:41 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 00:56:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 00:56:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:56:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:56:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:56:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:56:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:56:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:56:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:56:41 [DEBUG   ] Finished updating board flags in 1.191 seconds
2018-11-20 00:56:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 00:57:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 00:57:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 00:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:57:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364390748000000
2018-11-20 00:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:57:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364390748000000
2018-11-20 00:57:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 00:57:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:57:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:57:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:57:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:57:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:57:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:57:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 1
2018-11-20 00:57:41 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:57:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:57:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:57:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:57:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:57:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:57:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:57:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:57:41 [DEBUG   ] Finished updating board flags in 1.007 seconds
2018-11-20 00:57:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 00:58:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 00:58:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 00:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:58:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364402508000000
2018-11-20 00:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:58:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364402508000000
2018-11-20 00:58:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 00:58:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 00:58:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:58:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 00:58:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:58:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:58:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:58:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:58:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:58:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 00:58:41 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:58:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:58:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:58:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:58:41 [WARNING ] (13, 1, 'bot')
2018-11-20 00:58:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:58:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 00:58:41 [DEBUG   ] Finished updating board flags in 1.029 seconds
2018-11-20 00:58:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 00:59:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 00:59:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 00:59:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 00:59:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 00:59:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 00:59:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 00:59:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 00:59:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 00:59:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 00:59:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 00:59:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 00:59:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 00:59:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 00:59:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 00:59:42 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 00:59:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 00:59:42 [WARNING ] (13, 1, 'bot')
2018-11-20 00:59:42 [WARNING ] (13, 1, 'bot')
2018-11-20 00:59:42 [WARNING ] (13, 1, 'bot')
2018-11-20 00:59:42 [WARNING ] (13, 1, 'bot')
2018-11-20 00:59:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 00:59:42 [DEBUG   ] Finished updating board flags in 2.056 seconds
2018-11-20 00:59:44 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 01:00:04 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 01:00:04 [DEBUG   ] Got command RPT from MCS: ref #564228
2018-11-20 01:00:04 [DEBUG   ] summary = ERROR
2018-11-20 01:00:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564228  13 58442  3604249 A  ERRORERROR'
2018-11-20 01:00:04 [DEBUG   ] Got command RPT from MCS: ref #564229
2018-11-20 01:00:04 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:00:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564229  98 58442  3604441 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:00:04 [DEBUG   ] Got command RPT from MCS: ref #564230
2018-11-20 01:00:04 [DEBUG   ] summary = ERROR
2018-11-20 01:00:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564230  13 58442  3604645 A  ERRORERROR'
2018-11-20 01:00:04 [DEBUG   ] Got command RPT from MCS: ref #564232
2018-11-20 01:00:04 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:00:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564232  98 58442  3604849 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:00:05 [DEBUG   ] Got command RPT from MCS: ref #564234
2018-11-20 01:00:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:00:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564234  12 58442  3605041 A  ERRORB£ff'
2018-11-20 01:00:24 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:00:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:00:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364426224000000
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:00:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:00:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364426224000000
2018-11-20 01:00:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 01:00:41 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:00:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:00:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:00:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:00:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:00:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:00:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:00:41 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:00:41 [DEBUG   ] Finished updating board flags in 1.351 seconds
2018-11-20 01:00:44 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 01:00:46 [DEBUG   ] Got command RPT from MCS: ref #564238
2018-11-20 01:00:46 [DEBUG   ] summary = ERROR
2018-11-20 01:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564238  13 58442  3646553 A  ERRORERROR'
2018-11-20 01:00:46 [DEBUG   ] Got command RPT from MCS: ref #564239
2018-11-20 01:00:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564239  71 58442  3646554 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:01:04 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:01:25 [DEBUG   ] Finished updating ant stats in 1.402 seconds
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:01:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:01:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364437984000000
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:01:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:01:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364437984000000
2018-11-20 01:01:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 01:01:41 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:01:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:01:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:01:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:01:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:01:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:01:41 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:01:41 [DEBUG   ] Finished updating board flags in 1.234 seconds
2018-11-20 01:01:44 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:02:05 [DEBUG   ] Finished updating ant stats in 1.402 seconds
2018-11-20 01:02:24 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:02:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:02:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:02:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364449744000000
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:02:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:02:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364449744000000
2018-11-20 01:02:41 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:02:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:02:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:02:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:02:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:02:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:02:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:02:41 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:02:41 [DEBUG   ] Finished updating board flags in 1.179 seconds
2018-11-20 01:02:44 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 01:03:04 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:03:25 [DEBUG   ] Finished updating ant stats in 1.403 seconds
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:03:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:03:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:03:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364461504000000
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:03:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:03:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364461504000000
2018-11-20 01:03:41 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:03:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:03:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:03:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:03:41 [WARNING ] (13, 1, 'bot')
2018-11-20 01:03:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:03:41 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:03:41 [DEBUG   ] Finished updating board flags in 1.039 seconds
2018-11-20 01:03:44 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:04:05 [DEBUG   ] Finished updating ant stats in 1.402 seconds
2018-11-20 01:04:24 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:04:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:04:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:04:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:04:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:04:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:04:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:04:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:04:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:04:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:04:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:04:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:04:42 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:04:42 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:04:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:04:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:04:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:04:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:04:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:04:42 [DEBUG   ] Finished updating board flags in 1.546 seconds
2018-11-20 01:04:44 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 01:05:04 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:05:04 [DEBUG   ] Got command RPT from MCS: ref #564263
2018-11-20 01:05:04 [DEBUG   ] summary = ERROR
2018-11-20 01:05:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564263  13 58442  3904325 A  ERRORERROR'
2018-11-20 01:05:04 [DEBUG   ] Got command RPT from MCS: ref #564264
2018-11-20 01:05:04 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:05:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564264  98 58442  3904529 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:05:04 [DEBUG   ] Got command RPT from MCS: ref #564265
2018-11-20 01:05:04 [DEBUG   ] summary = ERROR
2018-11-20 01:05:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564265  13 58442  3904721 A  ERRORERROR'
2018-11-20 01:05:04 [DEBUG   ] Got command RPT from MCS: ref #564266
2018-11-20 01:05:04 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:05:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564266  98 58442  3904925 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:05:05 [DEBUG   ] Got command RPT from MCS: ref #564267
2018-11-20 01:05:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:05:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564267  12 58442  3905129 A  ERRORB¤ff'
2018-11-20 01:05:24 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:05:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:05:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364485024000000
2018-11-20 01:05:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:05:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:05:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:05:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:05:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364485024000000
2018-11-20 01:05:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:05:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:05:41 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:05:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:05:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:05:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:05:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:05:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:05:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:05:42 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:05:42 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:05:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:05:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:05:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:05:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:05:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:05:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:05:42 [DEBUG   ] Finished updating board flags in 1.389 seconds
2018-11-20 01:05:44 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:05:46 [DEBUG   ] Got command RPT from MCS: ref #564268
2018-11-20 01:05:46 [DEBUG   ] summary = ERROR
2018-11-20 01:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564268  13 58442  3946561 A  ERRORERROR'
2018-11-20 01:05:46 [DEBUG   ] Got command RPT from MCS: ref #564269
2018-11-20 01:05:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564269  71 58442  3946562 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:06:04 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:06:24 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 01:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:06:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364496784000000
2018-11-20 01:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:06:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364496784000000
2018-11-20 01:06:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 01:06:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:06:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:06:41 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 01:06:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:06:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:06:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:06:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:06:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:06:42 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:06:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:06:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:06:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:06:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:06:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:06:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:06:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:06:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:06:42 [DEBUG   ] Finished updating board flags in 1.281 seconds
2018-11-20 01:06:44 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:07:04 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:07:24 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:07:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364508544000000
2018-11-20 01:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:07:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364508544000000
2018-11-20 01:07:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 01:07:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:07:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:07:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:07:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:07:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:07:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:07:42 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:07:42 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:07:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:07:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:07:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:07:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:07:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:07:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:07:42 [DEBUG   ] Finished updating board flags in 1.114 seconds
2018-11-20 01:07:44 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:08:04 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:08:24 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:08:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364520304000000
2018-11-20 01:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:08:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364520304000000
2018-11-20 01:08:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 01:08:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:08:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:08:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:08:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:08:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:08:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:08:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:08:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:08:42 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:08:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:08:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:08:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:08:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:08:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:08:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:08:42 [DEBUG   ] Finished updating board flags in 1.018 seconds
2018-11-20 01:08:44 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-20 01:09:04 [DEBUG   ] Finished updating ant stats in 0.398 seconds
2018-11-20 01:09:25 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:09:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:09:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:09:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:09:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:09:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:09:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:09:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:09:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:09:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:09:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:09:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:09:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-20 01:09:43 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:09:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:09:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:09:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:09:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:09:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:09:43 [DEBUG   ] Finished updating board flags in 2.000 seconds
2018-11-20 01:09:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:10:04 [DEBUG   ] Got command RPT from MCS: ref #564291
2018-11-20 01:10:04 [DEBUG   ] summary = ERROR
2018-11-20 01:10:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564291  13 58442  4204397 A  ERRORERROR'
2018-11-20 01:10:04 [DEBUG   ] Got command RPT from MCS: ref #564292
2018-11-20 01:10:04 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:10:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564292  98 58442  4204601 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:10:04 [DEBUG   ] Got command RPT from MCS: ref #564293
2018-11-20 01:10:04 [DEBUG   ] summary = ERROR
2018-11-20 01:10:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564293  13 58442  4204805 A  ERRORERROR'
2018-11-20 01:10:05 [DEBUG   ] Got command RPT from MCS: ref #564295
2018-11-20 01:10:05 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:10:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564295  98 58442  4205009 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:10:05 [DEBUG   ] Got command RPT from MCS: ref #564297
2018-11-20 01:10:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:10:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564297  12 58442  4205201 A  ERRORB¤ff'
2018-11-20 01:10:05 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 01:10:25 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:10:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364543824000000
2018-11-20 01:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:10:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364543824000000
2018-11-20 01:10:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:10:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:10:41 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:10:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:10:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:10:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:10:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:10:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:10:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:10:42 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:10:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:10:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:10:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:10:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:10:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:10:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:10:42 [DEBUG   ] Finished updating board flags in 0.983 seconds
2018-11-20 01:10:45 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:10:46 [DEBUG   ] Got command RPT from MCS: ref #564305
2018-11-20 01:10:46 [DEBUG   ] summary = ERROR
2018-11-20 01:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564305  13 58442  4246557 A  ERRORERROR'
2018-11-20 01:10:46 [DEBUG   ] Got command RPT from MCS: ref #564306
2018-11-20 01:10:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564306  71 58442  4246558 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:11:05 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:11:25 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:11:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:11:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364555780000000
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:11:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:11:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364555780000000
2018-11-20 01:11:42 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:11:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:11:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:11:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:11:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:11:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:11:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:11:42 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:11:42 [DEBUG   ] Finished updating board flags in 1.273 seconds
2018-11-20 01:11:45 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 01:12:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:12:25 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:12:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:12:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:12:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364567540000000
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:12:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:12:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364567540000000
2018-11-20 01:12:42 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:12:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:12:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:12:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:12:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:12:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:12:42 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:12:42 [DEBUG   ] Finished updating board flags in 1.171 seconds
2018-11-20 01:12:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:13:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:13:25 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:13:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:13:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364579300000000
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:13:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:13:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364579300000000
2018-11-20 01:13:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 01:13:42 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:13:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:13:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:13:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:13:42 [WARNING ] (13, 1, 'bot')
2018-11-20 01:13:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:13:42 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:13:42 [DEBUG   ] Finished updating board flags in 1.102 seconds
2018-11-20 01:13:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:14:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:14:25 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 01:14:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:14:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:14:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:14:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:14:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:14:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 01:14:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:14:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 01:14:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:14:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:14:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:14:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:14:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:14:43 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:14:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:14:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:14:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:14:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:14:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:14:43 [DEBUG   ] Finished updating board flags in 1.543 seconds
2018-11-20 01:14:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:15:04 [DEBUG   ] Got command RPT from MCS: ref #564326
2018-11-20 01:15:04 [DEBUG   ] summary = ERROR
2018-11-20 01:15:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564326  13 58442  4504485 A  ERRORERROR'
2018-11-20 01:15:04 [DEBUG   ] Got command RPT from MCS: ref #564327
2018-11-20 01:15:04 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:15:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564327  98 58442  4504677 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:15:04 [DEBUG   ] Got command RPT from MCS: ref #564328
2018-11-20 01:15:04 [DEBUG   ] summary = ERROR
2018-11-20 01:15:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564328  13 58442  4504881 A  ERRORERROR'
2018-11-20 01:15:05 [DEBUG   ] Got command RPT from MCS: ref #564329
2018-11-20 01:15:05 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:15:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564329  98 58442  4505085 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:15:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:15:05 [DEBUG   ] Got command RPT from MCS: ref #564330
2018-11-20 01:15:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:15:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564330  12 58442  4505289 A  ERRORB¤ff'
2018-11-20 01:15:25 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 01:15:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:15:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364602820000000
2018-11-20 01:15:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:15:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:15:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:15:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:15:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364602820000000
2018-11-20 01:15:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 01:15:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:15:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:15:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:15:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:15:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:15:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:15:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:15:43 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:15:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:15:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:15:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:15:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:15:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:15:43 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:15:43 [DEBUG   ] Finished updating board flags in 1.342 seconds
2018-11-20 01:15:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:15:46 [DEBUG   ] Got command RPT from MCS: ref #564331
2018-11-20 01:15:46 [DEBUG   ] summary = ERROR
2018-11-20 01:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564331  13 58442  4546561 A  ERRORERROR'
2018-11-20 01:15:46 [DEBUG   ] Got command RPT from MCS: ref #564332
2018-11-20 01:15:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564332  71 58442  4546562 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:16:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:16:25 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:16:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:16:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364614580000000
2018-11-20 01:16:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:16:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:16:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:16:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:16:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364614580000000
2018-11-20 01:16:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:16:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:16:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:16:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:16:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:16:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:16:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:16:43 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:16:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:16:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:16:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:16:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:16:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:16:43 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:16:43 [DEBUG   ] Finished updating board flags in 1.196 seconds
2018-11-20 01:16:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:17:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:17:25 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:17:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:17:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364626340000000
2018-11-20 01:17:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:17:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:17:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:17:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:17:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364626340000000
2018-11-20 01:17:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:17:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:17:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:17:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:17:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:17:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:17:43 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:17:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:17:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:17:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:17:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:17:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:17:43 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:17:43 [DEBUG   ] Finished updating board flags in 0.991 seconds
2018-11-20 01:17:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:18:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:18:25 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:18:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:18:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:18:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:18:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:18:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:18:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 01:18:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:18:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:18:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:18:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:18:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:18:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:18:44 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:18:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:18:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:18:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:18:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:18:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:18:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:18:44 [DEBUG   ] Finished updating board flags in 1.840 seconds
2018-11-20 01:18:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:19:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:19:25 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:19:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 01:19:43 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 01:19:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:19:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364650056000000
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:19:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:19:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364650056000000
2018-11-20 01:19:43 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:19:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:19:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:19:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:19:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:19:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:19:43 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:19:43 [DEBUG   ] Finished updating board flags in 1.206 seconds
2018-11-20 01:19:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:20:04 [DEBUG   ] Got command RPT from MCS: ref #564358
2018-11-20 01:20:04 [DEBUG   ] summary = ERROR
2018-11-20 01:20:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564358  13 58442  4804557 A  ERRORERROR'
2018-11-20 01:20:04 [DEBUG   ] Got command RPT from MCS: ref #564359
2018-11-20 01:20:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:20:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564359  71 58442  4804761 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:20:04 [DEBUG   ] Got command RPT from MCS: ref #564360
2018-11-20 01:20:04 [DEBUG   ] summary = ERROR
2018-11-20 01:20:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564360  13 58442  4804965 A  ERRORERROR'
2018-11-20 01:20:05 [DEBUG   ] Got command RPT from MCS: ref #564362
2018-11-20 01:20:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:20:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564362  71 58442  4805169 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:20:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:20:05 [DEBUG   ] Got command RPT from MCS: ref #564363
2018-11-20 01:20:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:20:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564363  12 58442  4805361 A  ERRORB¤ff'
2018-11-20 01:20:25 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:20:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:20:43 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:20:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364661816000000
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:20:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:20:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364661816000000
2018-11-20 01:20:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:20:43 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:20:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:20:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:20:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:20:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:20:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:20:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:20:43 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:20:43 [DEBUG   ] Finished updating board flags in 1.276 seconds
2018-11-20 01:20:45 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 01:20:46 [DEBUG   ] Got command RPT from MCS: ref #564368
2018-11-20 01:20:46 [DEBUG   ] summary = ERROR
2018-11-20 01:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564368  13 58442  4846561 A  ERRORERROR'
2018-11-20 01:20:46 [DEBUG   ] Got command RPT from MCS: ref #564369
2018-11-20 01:20:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564369  71 58442  4846562 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:21:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:21:25 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:21:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 01:21:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:21:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:21:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364673576000000
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:21:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:21:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364673576000000
2018-11-20 01:21:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:21:43 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:21:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:21:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:21:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:21:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:21:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:21:43 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:21:43 [DEBUG   ] Finished updating board flags in 1.134 seconds
2018-11-20 01:21:45 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:22:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:22:25 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:22:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:22:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364685336000000
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:22:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:22:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364685336000000
2018-11-20 01:22:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:22:43 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:22:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:22:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:22:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:22:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:22:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:22:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:22:43 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:22:43 [DEBUG   ] Finished updating board flags in 1.047 seconds
2018-11-20 01:22:45 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:23:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:23:25 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:23:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:23:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364697096000000
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:23:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:23:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364697096000000
2018-11-20 01:23:43 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:23:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:23:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:23:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:23:43 [WARNING ] (13, 1, 'bot')
2018-11-20 01:23:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:23:43 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:23:43 [DEBUG   ] Finished updating board flags in 1.017 seconds
2018-11-20 01:23:45 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:24:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:24:25 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:24:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:24:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:24:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:24:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:24:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:24:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 01:24:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:24:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:24:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:24:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:24:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:24:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:24:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa5, chip 1
2018-11-20 01:24:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:24:44 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:24:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:24:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:24:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:24:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:24:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:24:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:24:44 [DEBUG   ] Finished updating board flags in 1.536 seconds
2018-11-20 01:24:45 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:25:04 [DEBUG   ] Got command RPT from MCS: ref #564389
2018-11-20 01:25:04 [DEBUG   ] summary = ERROR
2018-11-20 01:25:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564389  13 58442  5104649 A  ERRORERROR'
2018-11-20 01:25:04 [DEBUG   ] Got command RPT from MCS: ref #564390
2018-11-20 01:25:04 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:25:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564390  98 58442  5104841 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:25:05 [DEBUG   ] Got command RPT from MCS: ref #564391
2018-11-20 01:25:05 [DEBUG   ] summary = ERROR
2018-11-20 01:25:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564391  13 58442  5105045 A  ERRORERROR'
2018-11-20 01:25:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:25:05 [DEBUG   ] Got command RPT from MCS: ref #564392
2018-11-20 01:25:05 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:25:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564392  98 58442  5105249 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:25:05 [DEBUG   ] Got command RPT from MCS: ref #564393
2018-11-20 01:25:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:25:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564393  12 58442  5105441 A  ERRORB¤ff'
2018-11-20 01:25:25 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:25:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:25:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364720616000000
2018-11-20 01:25:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:25:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:25:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:25:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:25:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364720616000000
2018-11-20 01:25:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:25:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:25:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:25:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:25:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:25:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:25:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:25:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:25:44 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:25:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:25:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:25:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:25:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:25:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:25:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:25:44 [DEBUG   ] Finished updating board flags in 1.392 seconds
2018-11-20 01:25:45 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:25:46 [DEBUG   ] Got command RPT from MCS: ref #564398
2018-11-20 01:25:46 [DEBUG   ] summary = ERROR
2018-11-20 01:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564398  13 58442  5146557 A  ERRORERROR'
2018-11-20 01:25:46 [DEBUG   ] Got command RPT from MCS: ref #564399
2018-11-20 01:25:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564399  71 58442  5146558 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:26:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:26:25 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:26:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:26:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364732376000000
2018-11-20 01:26:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:26:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:26:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:26:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:26:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364732376000000
2018-11-20 01:26:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:26:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:26:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:26:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:26:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:26:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:26:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:26:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:26:44 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:26:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:26:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:26:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:26:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:26:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:26:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:26:44 [DEBUG   ] Finished updating board flags in 1.225 seconds
2018-11-20 01:26:45 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 01:27:05 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 01:27:25 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 01:27:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:27:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364744136000000
2018-11-20 01:27:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:27:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:27:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:27:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:27:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364744136000000
2018-11-20 01:27:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:27:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:27:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:27:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:27:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:27:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:27:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:27:44 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:27:44 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:27:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:27:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:27:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:27:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:27:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:27:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:27:44 [DEBUG   ] Finished updating board flags in 1.132 seconds
2018-11-20 01:27:45 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 01:28:05 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 01:28:25 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 01:28:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:28:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364755896000000
2018-11-20 01:28:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:28:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:28:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:28:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:28:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364755896000000
2018-11-20 01:28:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:28:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:28:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:28:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:28:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:28:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:28:44 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:28:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:28:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:28:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:28:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:28:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:28:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:28:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:28:44 [DEBUG   ] Finished updating board flags in 1.019 seconds
2018-11-20 01:28:45 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 01:29:05 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 01:29:25 [DEBUG   ] Finished updating ant stats in 0.398 seconds
2018-11-20 01:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:29:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364767656000000
2018-11-20 01:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:29:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364767656000000
2018-11-20 01:29:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:29:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:29:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:29:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:29:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:29:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:29:44 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:29:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:29:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:29:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:29:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:29:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:29:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:29:44 [DEBUG   ] Finished updating board flags in 0.989 seconds
2018-11-20 01:29:46 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 01:30:04 [DEBUG   ] Got command RPT from MCS: ref #564421
2018-11-20 01:30:04 [DEBUG   ] summary = ERROR
2018-11-20 01:30:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564421  13 58442  5404725 A  ERRORERROR'
2018-11-20 01:30:04 [DEBUG   ] Got command RPT from MCS: ref #564422
2018-11-20 01:30:04 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:30:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564422  71 58442  5404929 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:30:05 [DEBUG   ] Got command RPT from MCS: ref #564423
2018-11-20 01:30:05 [DEBUG   ] summary = ERROR
2018-11-20 01:30:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564423  13 58442  5405121 A  ERRORERROR'
2018-11-20 01:30:05 [DEBUG   ] Got command RPT from MCS: ref #564425
2018-11-20 01:30:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:30:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564425  71 58442  5405325 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:30:05 [DEBUG   ] Got command RPT from MCS: ref #564426
2018-11-20 01:30:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:30:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564426  12 58442  5405533 A  ERRORB¤ff'
2018-11-20 01:30:06 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 01:30:26 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 01:30:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:30:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:30:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:30:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:30:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:30:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:30:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:30:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:30:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:30:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:30:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:30:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:30:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:30:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:30:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:30:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:30:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:30:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:30:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:30:45 [DEBUG   ] Finished updating board flags in 1.848 seconds
2018-11-20 01:30:46 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:30:46 [DEBUG   ] Got command RPT from MCS: ref #564431
2018-11-20 01:30:46 [DEBUG   ] summary = ERROR
2018-11-20 01:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564431  13 58442  5446561 A  ERRORERROR'
2018-11-20 01:30:46 [DEBUG   ] Got command RPT from MCS: ref #564432
2018-11-20 01:30:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564432  98 58442  5446562 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:31:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:31:26 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:31:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:31:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:31:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364791372000000
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:31:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:31:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364791372000000
2018-11-20 01:31:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 01:31:44 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:31:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:31:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:31:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:31:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:31:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:31:44 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:31:44 [DEBUG   ] Finished updating board flags in 1.205 seconds
2018-11-20 01:31:46 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:32:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:32:26 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:32:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 01:32:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:32:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:32:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364803132000000
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:32:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:32:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364803132000000
2018-11-20 01:32:44 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:32:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:32:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:32:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:32:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:32:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:32:44 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:32:44 [DEBUG   ] Finished updating board flags in 1.267 seconds
2018-11-20 01:32:46 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:33:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:33:26 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:33:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:33:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:33:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:33:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:33:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:33:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:33:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:33:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:33:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:33:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:33:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:33:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:33:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:33:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:33:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:33:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:33:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:33:45 [DEBUG   ] Finished updating board flags in 1.765 seconds
2018-11-20 01:33:46 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 01:34:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:34:26 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:34:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:34:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364826652000000
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:34:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:34:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364826652000000
2018-11-20 01:34:44 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:34:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:34:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:34:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:34:44 [WARNING ] (13, 1, 'bot')
2018-11-20 01:34:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:34:44 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:34:44 [DEBUG   ] Finished updating board flags in 0.991 seconds
2018-11-20 01:34:46 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 01:35:04 [DEBUG   ] Got command RPT from MCS: ref #564456
2018-11-20 01:35:04 [DEBUG   ] summary = ERROR
2018-11-20 01:35:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564456  13 58442  5704801 A  ERRORERROR'
2018-11-20 01:35:05 [DEBUG   ] Got command RPT from MCS: ref #564457
2018-11-20 01:35:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:35:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564457  71 58442  5705005 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:35:05 [DEBUG   ] Got command RPT from MCS: ref #564458
2018-11-20 01:35:05 [DEBUG   ] summary = ERROR
2018-11-20 01:35:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564458  13 58442  5705209 A  ERRORERROR'
2018-11-20 01:35:05 [DEBUG   ] Got command RPT from MCS: ref #564459
2018-11-20 01:35:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:35:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564459  71 58442  5705413 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:35:05 [DEBUG   ] Got command RPT from MCS: ref #564460
2018-11-20 01:35:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:35:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564460  12 58442  5705605 A  ERRORB£ff'
2018-11-20 01:35:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:35:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:35:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:35:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364838412000000
2018-11-20 01:35:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:35:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:35:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:35:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:35:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364838412000000
2018-11-20 01:35:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:35:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:35:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:35:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:35:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:35:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:35:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 01:35:45 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:35:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:35:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:35:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:35:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:35:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:35:45 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:35:45 [DEBUG   ] Finished updating board flags in 1.365 seconds
2018-11-20 01:35:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:35:46 [DEBUG   ] Got command RPT from MCS: ref #564461
2018-11-20 01:35:46 [DEBUG   ] summary = ERROR
2018-11-20 01:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564461  13 58442  5746565 A  ERRORERROR'
2018-11-20 01:35:46 [DEBUG   ] Got command RPT from MCS: ref #564462
2018-11-20 01:35:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564462  71 58442  5746566 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:36:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:36:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:36:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:36:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364850172000000
2018-11-20 01:36:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:36:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:36:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:36:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:36:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364850172000000
2018-11-20 01:36:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-20 01:36:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:36:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:36:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:36:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:36:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:36:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:36:45 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:36:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:36:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:36:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:36:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:36:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:36:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:36:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:36:45 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:36:45 [DEBUG   ] Finished updating board flags in 1.225 seconds
2018-11-20 01:36:46 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 01:37:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:37:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:37:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:37:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364861932000000
2018-11-20 01:37:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:37:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:37:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:37:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:37:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364861932000000
2018-11-20 01:37:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:37:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:37:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:37:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:37:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:37:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:37:45 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:37:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:37:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:37:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:37:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:37:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:37:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:37:45 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:37:45 [DEBUG   ] Finished updating board flags in 1.140 seconds
2018-11-20 01:37:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:38:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:38:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:38:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:38:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364873692000000
2018-11-20 01:38:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:38:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:38:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:38:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:38:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364873692000000
2018-11-20 01:38:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:38:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:38:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:38:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:38:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:38:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:38:45 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:38:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:38:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:38:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:38:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:38:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:38:45 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:38:45 [DEBUG   ] Finished updating board flags in 1.027 seconds
2018-11-20 01:38:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:39:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:39:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:39:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:39:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:39:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:39:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:39:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:39:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 01:39:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:39:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:39:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:39:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:39:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:39:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:39:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:39:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:39:46 [DEBUG   ] Finished updating ant stats in 0.831 seconds
2018-11-20 01:39:46 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:39:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:39:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:39:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:39:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:39:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:39:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:39:46 [DEBUG   ] Finished updating board flags in 2.049 seconds
2018-11-20 01:40:04 [DEBUG   ] Got command RPT from MCS: ref #564484
2018-11-20 01:40:04 [DEBUG   ] summary = ERROR
2018-11-20 01:40:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564484  13 58442  6004893 A  ERRORERROR'
2018-11-20 01:40:05 [DEBUG   ] Got command RPT from MCS: ref #564485
2018-11-20 01:40:05 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:40:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564485  98 58442  6005085 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:40:05 [DEBUG   ] Got command RPT from MCS: ref #564486
2018-11-20 01:40:05 [DEBUG   ] summary = ERROR
2018-11-20 01:40:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564486  13 58442  6005289 A  ERRORERROR'
2018-11-20 01:40:05 [DEBUG   ] Got command RPT from MCS: ref #564488
2018-11-20 01:40:05 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 01:40:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564488  98 58442  6005494 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 01:40:05 [DEBUG   ] Got command RPT from MCS: ref #564489
2018-11-20 01:40:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:40:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564489  12 58442  6005694 A  ERRORB¥ff'
2018-11-20 01:40:06 [DEBUG   ] Finished updating ant stats in 0.774 seconds
2018-11-20 01:40:26 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:40:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 01:40:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:40:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:40:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364897408000000
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:40:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:40:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364897408000000
2018-11-20 01:40:45 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:40:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:40:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:40:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:40:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:40:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:40:45 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:40:45 [DEBUG   ] Finished updating board flags in 1.424 seconds
2018-11-20 01:40:46 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 01:40:46 [DEBUG   ] Got command RPT from MCS: ref #564498
2018-11-20 01:40:46 [DEBUG   ] summary = ERROR
2018-11-20 01:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564498  13 58442  6046569 A  ERRORERROR'
2018-11-20 01:40:46 [DEBUG   ] Got command RPT from MCS: ref #564499
2018-11-20 01:40:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564499  71 58442  6046570 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:41:06 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 01:41:26 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:41:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 01:41:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:41:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:41:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:41:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364909168000000
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:41:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:41:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364909168000000
2018-11-20 01:41:45 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:41:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:41:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:41:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:41:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:41:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:41:45 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:41:45 [DEBUG   ] Finished updating board flags in 1.431 seconds
2018-11-20 01:41:46 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 01:42:06 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:42:26 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:42:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:42:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:42:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:42:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364920928000000
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:42:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:42:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364920928000000
2018-11-20 01:42:45 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:42:46 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-20 01:42:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:42:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:42:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:42:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:42:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:42:46 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:42:46 [DEBUG   ] Finished updating board flags in 1.724 seconds
2018-11-20 01:43:06 [DEBUG   ] Finished updating ant stats in 0.398 seconds
2018-11-20 01:43:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:43:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:43:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364932688000000
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:43:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:43:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364932688000000
2018-11-20 01:43:45 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:43:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 01:43:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:43:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:43:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:43:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:43:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:43:45 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:43:45 [DEBUG   ] Finished updating board flags in 1.199 seconds
2018-11-20 01:43:47 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:44:07 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:44:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:44:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:44:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:44:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364944448000000
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:44:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:44:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364944448000000
2018-11-20 01:44:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 01:44:45 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:44:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:44:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:44:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:44:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:44:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:44:45 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:44:45 [DEBUG   ] Finished updating board flags in 1.035 seconds
2018-11-20 01:44:47 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:45:04 [DEBUG   ] Got command RPT from MCS: ref #564519
2018-11-20 01:45:04 [DEBUG   ] summary = ERROR
2018-11-20 01:45:04 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564519  13 58442  6304965 A  ERRORERROR'
2018-11-20 01:45:05 [DEBUG   ] Got command RPT from MCS: ref #564520
2018-11-20 01:45:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:45:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564520  71 58442  6305169 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:45:05 [DEBUG   ] Got command RPT from MCS: ref #564521
2018-11-20 01:45:05 [DEBUG   ] summary = ERROR
2018-11-20 01:45:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564521  13 58442  6305373 A  ERRORERROR'
2018-11-20 01:45:05 [DEBUG   ] Got command RPT from MCS: ref #564522
2018-11-20 01:45:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:45:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564522  71 58442  6305577 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:45:05 [DEBUG   ] Got command RPT from MCS: ref #564523
2018-11-20 01:45:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:45:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564523  12 58442  6305769 A  ERRORB¤ff'
2018-11-20 01:45:07 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:45:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:45:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 01:45:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:45:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:45:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302364956208000000
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:45:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:45:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302364956208000000
2018-11-20 01:45:45 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:45:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:45:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:45:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:45:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:45:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:45:45 [WARNING ] (13, 1, 'bot')
2018-11-20 01:45:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:45:45 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:45:45 [DEBUG   ] Finished updating board flags in 0.966 seconds
2018-11-20 01:45:46 [DEBUG   ] Got command RPT from MCS: ref #564524
2018-11-20 01:45:46 [DEBUG   ] summary = ERROR
2018-11-20 01:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564524  13 58442  6346569 A  ERRORERROR'
2018-11-20 01:45:46 [DEBUG   ] Got command RPT from MCS: ref #564525
2018-11-20 01:45:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564525  71 58442  6346570 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:45:47 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:46:07 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:46:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:46:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:46:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364967968000000
2018-11-20 01:46:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:46:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:46:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:46:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:46:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364967968000000
2018-11-20 01:46:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:46:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:46:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:46:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:46:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:46:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:46:46 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:46:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:46:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:46:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:46:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:46:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:46:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:46:46 [DEBUG   ] Finished updating board flags in 1.401 seconds
2018-11-20 01:46:47 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:47:07 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:47:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:47:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:47:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364979728000000
2018-11-20 01:47:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:47:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:47:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:47:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:47:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364979728000000
2018-11-20 01:47:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:47:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:47:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:47:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:47:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:47:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:47:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:47:46 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:47:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:47:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:47:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:47:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:47:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:47:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:47:46 [DEBUG   ] Finished updating board flags in 1.455 seconds
2018-11-20 01:47:47 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:48:07 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:48:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:48:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:48:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302364991488000000
2018-11-20 01:48:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:48:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:48:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:48:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:48:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302364991488000000
2018-11-20 01:48:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:48:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:48:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:48:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:48:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:48:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:48:46 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:48:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:48:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:48:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:48:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:48:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:48:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:48:46 [DEBUG   ] Finished updating board flags in 1.376 seconds
2018-11-20 01:48:47 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:49:07 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:49:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:49:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:49:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365003248000000
2018-11-20 01:49:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:49:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:49:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:49:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:49:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365003248000000
2018-11-20 01:49:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 01:49:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:49:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:49:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:49:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:49:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:49:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:49:47 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 01:49:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-20 01:49:47 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:49:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:49:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:49:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:49:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:49:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:49:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:49:47 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:49:47 [DEBUG   ] Finished updating board flags in 2.302 seconds
2018-11-20 01:50:05 [DEBUG   ] Got command RPT from MCS: ref #564551
2018-11-20 01:50:05 [DEBUG   ] summary = ERROR
2018-11-20 01:50:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564551  13 58442  6605049 A  ERRORERROR'
2018-11-20 01:50:05 [DEBUG   ] Got command RPT from MCS: ref #564552
2018-11-20 01:50:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:50:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564552  71 58442  6605253 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:50:05 [DEBUG   ] Got command RPT from MCS: ref #564553
2018-11-20 01:50:05 [DEBUG   ] summary = ERROR
2018-11-20 01:50:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564553  13 58442  6605457 A  ERRORERROR'
2018-11-20 01:50:05 [DEBUG   ] Got command RPT from MCS: ref #564555
2018-11-20 01:50:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:50:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564555  71 58442  6605649 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:50:05 [DEBUG   ] Got command RPT from MCS: ref #564556
2018-11-20 01:50:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:50:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564556  12 58442  6605853 A  ERRORB£ff'
2018-11-20 01:50:07 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 01:50:27 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:50:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:50:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365015008000000
2018-11-20 01:50:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:50:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:50:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:50:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:50:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365015008000000
2018-11-20 01:50:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 01:50:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:50:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 01:50:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:50:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:50:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:50:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:50:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:50:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:50:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 01:50:46 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:50:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:50:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:50:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:50:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:50:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:50:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:50:46 [DEBUG   ] Finished updating board flags in 1.079 seconds
2018-11-20 01:50:46 [DEBUG   ] Got command RPT from MCS: ref #564561
2018-11-20 01:50:46 [DEBUG   ] summary = ERROR
2018-11-20 01:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564561  13 58442  6646569 A  ERRORERROR'
2018-11-20 01:50:46 [DEBUG   ] Got command RPT from MCS: ref #564562
2018-11-20 01:50:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 01:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564562  71 58442  6646570 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 01:50:47 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:51:07 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:51:27 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 01:51:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:51:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:51:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:51:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:51:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:51:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 01:51:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:51:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:51:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:51:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:51:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:51:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:51:47 [DEBUG   ] Finished updating ant stats in 1.028 seconds
2018-11-20 01:51:47 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:51:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:51:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:51:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:51:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:51:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:51:47 [DEBUG   ] Finished updating board flags in 2.044 seconds
2018-11-20 01:52:07 [DEBUG   ] Finished updating ant stats in 0.974 seconds
2018-11-20 01:52:27 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 01:52:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:52:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:52:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:52:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:52:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:52:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:52:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:52:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:52:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:52:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:52:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:52:47 [DEBUG   ] Finished updating ant stats in 0.828 seconds
2018-11-20 01:52:47 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:52:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:52:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:52:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:52:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:52:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:52:47 [DEBUG   ] Finished updating board flags in 1.975 seconds
2018-11-20 01:53:07 [DEBUG   ] Finished updating ant stats in 0.774 seconds
2018-11-20 01:53:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:53:46 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:53:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:53:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:53:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365050484000000
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:53:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:53:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365050484000000
2018-11-20 01:53:47 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 01:53:47 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:53:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:53:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:53:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:53:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:53:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:53:47 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:53:47 [DEBUG   ] Finished updating board flags in 1.752 seconds
2018-11-20 01:54:07 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 01:54:27 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:54:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 01:54:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 01:54:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:54:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365062244000000
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:54:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:54:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365062244000000
2018-11-20 01:54:46 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 01:54:46 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 01:54:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:54:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:54:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:54:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:54:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:54:46 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:54:46 [DEBUG   ] Finished updating board flags in 1.111 seconds
2018-11-20 01:54:47 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:55:05 [DEBUG   ] Got command RPT from MCS: ref #564582
2018-11-20 01:55:05 [DEBUG   ] summary = ERROR
2018-11-20 01:55:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564582  13 58442  6905129 A  ERRORERROR'
2018-11-20 01:55:05 [DEBUG   ] Got command RPT from MCS: ref #564583
2018-11-20 01:55:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:55:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564583  71 58442  6905325 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:55:05 [DEBUG   ] Got command RPT from MCS: ref #564584
2018-11-20 01:55:05 [DEBUG   ] summary = ERROR
2018-11-20 01:55:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564584  13 58442  6905529 A  ERRORERROR'
2018-11-20 01:55:05 [DEBUG   ] Got command RPT from MCS: ref #564585
2018-11-20 01:55:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:55:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564585  71 58442  6905733 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:55:05 [DEBUG   ] Got command RPT from MCS: ref #564586
2018-11-20 01:55:05 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 01:55:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564586  12 58442  6905937 A  ERRORB£ff'
2018-11-20 01:55:07 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:55:27 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:55:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 01:55:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:55:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365074004000000
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:55:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:55:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365074004000000
2018-11-20 01:55:46 [DEBUG   ] Got command RPT from MCS: ref #564591
2018-11-20 01:55:46 [DEBUG   ] summary = ERROR
2018-11-20 01:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564591  13 58442  6946563 A  ERRORERROR'
2018-11-20 01:55:46 [DEBUG   ] Got command RPT from MCS: ref #564592
2018-11-20 01:55:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1
2018-11-20 01:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564592  71 58442  6946563 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #2, #1'
2018-11-20 01:55:46 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:55:46 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 01:55:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:55:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:55:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:55:46 [WARNING ] (13, 1, 'bot')
2018-11-20 01:55:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:55:46 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 01:55:46 [DEBUG   ] Finished updating board flags in 0.940 seconds
2018-11-20 01:55:47 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:56:07 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:56:27 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 01:56:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:56:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365085764000000
2018-11-20 01:56:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:56:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:56:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:56:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:56:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365085764000000
2018-11-20 01:56:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:56:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:56:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:56:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:56:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:56:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:56:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:56:47 [DEBUG   ] Finished updating ant stats in 0.430 seconds
2018-11-20 01:56:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:56:47 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:56:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:56:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:56:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:56:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:56:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:56:47 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:56:47 [DEBUG   ] Finished updating board flags in 1.438 seconds
2018-11-20 01:57:07 [DEBUG   ] Finished updating ant stats in 0.374 seconds
2018-11-20 01:57:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:57:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:57:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365097524000000
2018-11-20 01:57:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:57:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:57:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:57:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:57:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365097524000000
2018-11-20 01:57:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 01:57:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:57:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 01:57:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:57:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:57:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:57:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:57:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:57:47 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:57:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:57:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:57:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:57:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:57:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:57:47 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:57:47 [DEBUG   ] Finished updating board flags in 1.342 seconds
2018-11-20 01:57:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:58:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:58:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:58:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:58:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365109284000000
2018-11-20 01:58:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:58:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:58:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:58:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:58:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365109284000000
2018-11-20 01:58:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 01:58:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:58:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:58:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:58:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:58:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:58:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:58:47 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:58:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:58:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:58:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:58:47 [WARNING ] (13, 1, 'bot')
2018-11-20 01:58:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:58:47 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:58:47 [DEBUG   ] Finished updating board flags in 1.235 seconds
2018-11-20 01:58:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:59:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:59:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 01:59:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 01:59:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365121044000000
2018-11-20 01:59:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 01:59:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 01:59:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 01:59:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 01:59:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365121044000000
2018-11-20 01:59:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 01:59:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 01:59:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 01:59:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 01:59:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 01:59:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 01:59:48 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 01:59:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 01:59:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 01:59:48 [WARNING ] (13, 1, 'bot')
2018-11-20 01:59:48 [WARNING ] (13, 1, 'bot')
2018-11-20 01:59:48 [WARNING ] (13, 1, 'bot')
2018-11-20 01:59:48 [WARNING ] (13, 1, 'bot')
2018-11-20 01:59:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 01:59:48 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 01:59:48 [DEBUG   ] Finished updating board flags in 2.324 seconds
2018-11-20 02:00:05 [DEBUG   ] Got command RPT from MCS: ref #564614
2018-11-20 02:00:05 [DEBUG   ] summary = ERROR
2018-11-20 02:00:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564614  13 58442  7205209 A  ERRORERROR'
2018-11-20 02:00:05 [DEBUG   ] Got command RPT from MCS: ref #564615
2018-11-20 02:00:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 02:00:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564615  71 58442  7205413 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 02:00:05 [DEBUG   ] Got command RPT from MCS: ref #564616
2018-11-20 02:00:05 [DEBUG   ] summary = ERROR
2018-11-20 02:00:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564616  13 58442  7205617 A  ERRORERROR'
2018-11-20 02:00:05 [DEBUG   ] Got command RPT from MCS: ref #564618
2018-11-20 02:00:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 02:00:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564618  71 58442  7205809 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 02:00:06 [DEBUG   ] Got command RPT from MCS: ref #564620
2018-11-20 02:00:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:00:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564620  12 58442  7206013 A  ERRORB¤ff'
2018-11-20 02:00:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:00:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:00:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:00:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365132804000000
2018-11-20 02:00:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:00:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:00:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:00:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:00:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365132804000000
2018-11-20 02:00:46 [DEBUG   ] Got command RPT from MCS: ref #564624
2018-11-20 02:00:46 [DEBUG   ] summary = ERROR
2018-11-20 02:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564624  13 58442  7246565 A  ERRORERROR'
2018-11-20 02:00:46 [DEBUG   ] Got command RPT from MCS: ref #564625
2018-11-20 02:00:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 02:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564625  71 58442  7246566 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 02:00:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:00:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:00:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:00:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:00:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:00:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:00:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 02:00:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 02:00:47 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:00:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:00:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:00:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:00:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:00:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:00:47 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:00:47 [DEBUG   ] Finished updating board flags in 1.046 seconds
2018-11-20 02:00:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:01:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:01:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:01:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:01:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:01:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:01:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:01:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:01:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:01:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:01:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:01:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:01:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:01:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:01:48 [DEBUG   ] Finished updating ant stats in 1.232 seconds
2018-11-20 02:01:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:01:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:01:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:01:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:01:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:01:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:01:48 [DEBUG   ] Finished updating board flags in 2.065 seconds
2018-11-20 02:02:08 [DEBUG   ] Finished updating ant stats in 1.170 seconds
2018-11-20 02:02:28 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:02:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:02:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:02:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:02:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:02:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:02:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:02:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:02:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:02:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:02:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:02:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:02:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 02:02:48 [DEBUG   ] Finished updating ant stats in 1.029 seconds
2018-11-20 02:02:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:02:48 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:02:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:02:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:02:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:02:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:02:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:02:48 [DEBUG   ] Finished updating board flags in 1.970 seconds
2018-11-20 02:03:08 [DEBUG   ] Finished updating ant stats in 0.974 seconds
2018-11-20 02:03:28 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:03:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:03:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:03:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:03:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:03:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:03:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:03:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:03:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:03:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:03:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:03:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:03:48 [DEBUG   ] Finished updating ant stats in 0.830 seconds
2018-11-20 02:03:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:03:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:03:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:03:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:03:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:03:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:03:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:03:48 [DEBUG   ] Finished updating board flags in 1.809 seconds
2018-11-20 02:04:08 [DEBUG   ] Finished updating ant stats in 0.774 seconds
2018-11-20 02:04:28 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:04:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:04:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:04:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:04:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:04:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:04:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:04:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:04:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:04:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:04:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:04:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:04:48 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 02:04:48 [DEBUG   ] Finished updating ant stats in 0.628 seconds
2018-11-20 02:04:48 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:04:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:04:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:04:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:04:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:04:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:04:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:04:48 [DEBUG   ] Finished updating board flags in 1.773 seconds
2018-11-20 02:05:05 [DEBUG   ] Got command RPT from MCS: ref #564649
2018-11-20 02:05:05 [DEBUG   ] summary = ERROR
2018-11-20 02:05:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564649  13 58442  7505289 A  ERRORERROR'
2018-11-20 02:05:05 [DEBUG   ] Got command RPT from MCS: ref #564650
2018-11-20 02:05:05 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:05:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564650  98 58442  7505493 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:05:05 [DEBUG   ] Got command RPT from MCS: ref #564651
2018-11-20 02:05:05 [DEBUG   ] summary = ERROR
2018-11-20 02:05:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564651  13 58442  7505689 A  ERRORERROR'
2018-11-20 02:05:05 [DEBUG   ] Got command RPT from MCS: ref #564652
2018-11-20 02:05:05 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:05:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564652  98 58442  7505897 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:05:06 [DEBUG   ] Got command RPT from MCS: ref #564653
2018-11-20 02:05:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:05:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564653  12 58442  7506097 A  ERRORB£ff'
2018-11-20 02:05:08 [DEBUG   ] Finished updating ant stats in 0.578 seconds
2018-11-20 02:05:28 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 02:05:46 [DEBUG   ] Got command RPT from MCS: ref #564654
2018-11-20 02:05:46 [DEBUG   ] summary = ERROR
2018-11-20 02:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564654  13 58442  7546565 A  ERRORERROR'
2018-11-20 02:05:46 [DEBUG   ] Got command RPT from MCS: ref #564655
2018-11-20 02:05:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564655  98 58442  7546566 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:05:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:05:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365191800000000
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:05:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:05:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365191800000000
2018-11-20 02:05:47 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:05:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:05:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:05:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:05:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:05:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:05:47 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 02:05:47 [DEBUG   ] Finished updating board flags in 1.038 seconds
2018-11-20 02:05:48 [DEBUG   ] Finished updating ant stats in 0.398 seconds
2018-11-20 02:06:09 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:06:29 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:06:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 02:06:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:06:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365203560000000
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:06:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:06:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365203560000000
2018-11-20 02:06:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 02:06:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-20 02:06:47 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:06:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:06:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:06:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:06:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:06:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:06:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:06:47 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 02:06:47 [DEBUG   ] Finished updating board flags in 1.013 seconds
2018-11-20 02:06:49 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 02:07:09 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:07:29 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:07:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 02:07:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:07:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365215320000000
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:07:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:07:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365215320000000
2018-11-20 02:07:47 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:07:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:07:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:07:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:07:47 [WARNING ] (13, 1, 'bot')
2018-11-20 02:07:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:07:47 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 02:07:47 [DEBUG   ] Finished updating board flags in 0.952 seconds
2018-11-20 02:07:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:08:09 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:08:29 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:08:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:08:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365227080000000
2018-11-20 02:08:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:08:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:08:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:08:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:08:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365227080000000
2018-11-20 02:08:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:08:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:08:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:08:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:08:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:08:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:08:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:08:48 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:08:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:08:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:08:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:08:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:08:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:08:48 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:08:48 [DEBUG   ] Finished updating board flags in 1.416 seconds
2018-11-20 02:08:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:09:09 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 02:09:29 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:09:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:09:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365238840000000
2018-11-20 02:09:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:09:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:09:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:09:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:09:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365238840000000
2018-11-20 02:09:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 02:09:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 02:09:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 02:09:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:09:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:09:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:09:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:09:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:09:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:09:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 02:09:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:09:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:09:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:09:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:09:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:09:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:09:48 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:09:48 [DEBUG   ] Finished updating board flags in 1.348 seconds
2018-11-20 02:09:49 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:10:05 [DEBUG   ] Got command RPT from MCS: ref #564677
2018-11-20 02:10:05 [DEBUG   ] summary = ERROR
2018-11-20 02:10:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564677  13 58442  7805369 A  ERRORERROR'
2018-11-20 02:10:05 [DEBUG   ] Got command RPT from MCS: ref #564678
2018-11-20 02:10:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 02:10:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564678  71 58442  7805573 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 02:10:05 [DEBUG   ] Got command RPT from MCS: ref #564679
2018-11-20 02:10:05 [DEBUG   ] summary = ERROR
2018-11-20 02:10:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564679  13 58442  7805777 A  ERRORERROR'
2018-11-20 02:10:05 [DEBUG   ] Got command RPT from MCS: ref #564681
2018-11-20 02:10:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 02:10:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564681  71 58442  7805981 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 02:10:06 [DEBUG   ] Got command RPT from MCS: ref #564682
2018-11-20 02:10:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:10:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564682  12 58442  7806173 A  ERRORB¤ff'
2018-11-20 02:10:09 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:10:29 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:10:46 [DEBUG   ] Got command RPT from MCS: ref #564691
2018-11-20 02:10:46 [DEBUG   ] summary = ERROR
2018-11-20 02:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564691  13 58442  7846569 A  ERRORERROR'
2018-11-20 02:10:46 [DEBUG   ] Got command RPT from MCS: ref #564692
2018-11-20 02:10:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 02:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564692  71 58442  7846570 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 02:10:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:10:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365250600000000
2018-11-20 02:10:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:10:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:10:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:10:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:10:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365250600000000
2018-11-20 02:10:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:10:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:10:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:10:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:10:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:10:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:10:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:10:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:10:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 02:10:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:10:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:10:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:10:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:10:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:10:48 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:10:48 [DEBUG   ] Finished updating board flags in 1.255 seconds
2018-11-20 02:10:49 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:11:09 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:11:29 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:11:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:11:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365262360000000
2018-11-20 02:11:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:11:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:11:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:11:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:11:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365262360000000
2018-11-20 02:11:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 02:11:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 02:11:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 02:11:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:11:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:11:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:11:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:11:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:11:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:11:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:11:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:11:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:11:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:11:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:11:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:11:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:11:48 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:11:48 [DEBUG   ] Finished updating board flags in 1.213 seconds
2018-11-20 02:11:49 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 02:12:09 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:12:29 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:12:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:12:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365274120000000
2018-11-20 02:12:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:12:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:12:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:12:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:12:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365274120000000
2018-11-20 02:12:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-20 02:12:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 02:12:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 02:12:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:12:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:12:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:12:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:12:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:12:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:12:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:12:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:12:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:12:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:12:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:12:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:12:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:12:48 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:12:48 [DEBUG   ] Finished updating board flags in 1.143 seconds
2018-11-20 02:12:49 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:13:09 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:13:29 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:13:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:13:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365285880000000
2018-11-20 02:13:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:13:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:13:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:13:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:13:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365285880000000
2018-11-20 02:13:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 02:13:47 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 02:13:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 02:13:47 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:13:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:13:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:13:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:13:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:13:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:13:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:13:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:13:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:13:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:13:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:13:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:13:48 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:13:48 [DEBUG   ] Finished updating board flags in 1.004 seconds
2018-11-20 02:13:49 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:14:00 [DEBUG   ] Got command TBN from MCS: ref #564712
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa27', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '1', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa24', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '2', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] tbn = exited with status True, code 0x00
2018-11-20 02:14:00 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_TBN   564712   8 58442  8040961 A  ERROR'
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa23', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '3', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa18', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '4', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa10', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '5', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa32', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '6', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa11', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '7', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa30', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '8', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa12', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '9', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa8', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '10', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa13', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '11', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa26', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '12', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa14', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '13', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa22', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '14', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa20', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '15', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa1', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '16', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa5', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '17', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa25', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '18', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa35', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '19', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa19', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '20', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa21', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '21', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa6', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '22', '1542680042']
2018-11-20 02:14:00 [DEBUG   ] spawn_process: at 2018-11-20 02:14:00 running ['/usr/local/bin/lwa7', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '23', '1542680042']
2018-11-20 02:14:01 [DEBUG   ] spawn_process: at 2018-11-20 02:14:01 running ['/usr/local/bin/lwa33', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '24', '1542680042']
2018-11-20 02:14:01 [DEBUG   ] spawn_process: at 2018-11-20 02:14:01 running ['/usr/local/bin/lwa15', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '25', '1542680042']
2018-11-20 02:14:01 [DEBUG   ] spawn_process: at 2018-11-20 02:14:01 running ['/usr/local/bin/lwa28', '/lwa/software/tbn', '192.168.0.37', '10000', '38100000.0', '7', '20', '0', '26', '1542680042']
2018-11-20 02:14:01 [INFO    ] Finished spawning TBN in 0.049 s
2018-11-20 02:14:09 [DEBUG   ] Finished updating ant stats in 0.671 seconds
2018-11-20 02:14:29 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 02:14:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:14:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365297640000000
2018-11-20 02:14:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:14:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:14:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:14:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:14:47 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365297640000000
2018-11-20 02:14:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 02:14:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:14:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:14:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365297836000000
2018-11-20 02:14:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:14:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:14:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:14:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:14:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365297836000000
2018-11-20 02:14:48 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:14:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:14:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:14:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:14:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:14:48 [WARNING ] (13, 1, 'bot')
2018-11-20 02:14:48 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:14:48 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 02:14:48 [DEBUG   ] Finished updating board flags in 1.464 seconds
2018-11-20 02:14:49 [DEBUG   ] Finished updating ant stats in 0.597 seconds
2018-11-20 02:15:05 [DEBUG   ] Got command RPT from MCS: ref #564713
2018-11-20 02:15:05 [DEBUG   ] summary = ERROR
2018-11-20 02:15:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564713  13 58442  8105461 A  ERRORERROR'
2018-11-20 02:15:05 [DEBUG   ] Got command RPT from MCS: ref #564714
2018-11-20 02:15:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:15:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564714  79 58442  8105653 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:15:05 [DEBUG   ] Got command RPT from MCS: ref #564715
2018-11-20 02:15:05 [DEBUG   ] summary = ERROR
2018-11-20 02:15:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564715  13 58442  8105857 A  ERRORERROR'
2018-11-20 02:15:06 [DEBUG   ] Got command RPT from MCS: ref #564716
2018-11-20 02:15:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:15:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564716  79 58442  8106061 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:15:06 [DEBUG   ] Got command RPT from MCS: ref #564717
2018-11-20 02:15:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:15:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564717  12 58442  8106253 A  ERRORB¤ff'
2018-11-20 02:15:10 [DEBUG   ] Finished updating ant stats in 1.395 seconds
2018-11-20 02:15:30 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 02:15:46 [DEBUG   ] Got command RPT from MCS: ref #564718
2018-11-20 02:15:46 [DEBUG   ] summary = ERROR
2018-11-20 02:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564718  13 58442  8146573 A  ERRORERROR'
2018-11-20 02:15:46 [DEBUG   ] Got command RPT from MCS: ref #564719
2018-11-20 02:15:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564719  79 58442  8146574 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:15:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:15:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:15:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:15:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:15:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:15:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:15:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:15:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:15:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:15:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:15:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:15:49 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:15:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:15:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:15:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:15:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:15:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:15:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:15:49 [DEBUG   ] Finished updating board flags in 2.205 seconds
2018-11-20 02:15:50 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:16:10 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:16:30 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 02:16:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:16:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:16:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:16:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:16:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:16:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 02:16:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:16:48 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:16:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 02:16:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:16:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:16:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:16:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:16:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:16:49 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:16:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:16:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:16:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:16:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:16:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:16:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:16:49 [DEBUG   ] Finished updating board flags in 2.182 seconds
2018-11-20 02:16:50 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-20 02:17:10 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:17:30 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 02:17:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:17:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:17:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:17:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:17:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:17:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:17:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:17:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:17:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:17:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:17:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:17:49 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:17:49 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:17:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:17:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:17:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:17:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:17:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:17:49 [DEBUG   ] Finished updating board flags in 2.019 seconds
2018-11-20 02:17:50 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 02:18:10 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 02:18:30 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:18:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:18:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:18:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:18:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:18:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:18:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:18:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:18:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:18:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:18:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:18:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:18:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:18:49 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:18:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:18:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:18:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:18:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:18:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:18:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:18:49 [DEBUG   ] Finished updating board flags in 1.984 seconds
2018-11-20 02:18:50 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 02:19:10 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 02:19:30 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:19:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:19:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:19:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:19:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:19:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:19:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 02:19:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:19:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:19:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:19:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:19:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:19:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:19:49 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:19:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:19:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:19:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:19:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:19:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:19:49 [DEBUG   ] Finished updating board flags in 1.867 seconds
2018-11-20 02:19:50 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 02:20:05 [DEBUG   ] Got command RPT from MCS: ref #564745
2018-11-20 02:20:05 [DEBUG   ] summary = ERROR
2018-11-20 02:20:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564745  13 58442  8405533 A  ERRORERROR'
2018-11-20 02:20:05 [DEBUG   ] Got command RPT from MCS: ref #564746
2018-11-20 02:20:05 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:20:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564746  98 58442  8405737 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:20:05 [DEBUG   ] Got command RPT from MCS: ref #564747
2018-11-20 02:20:05 [DEBUG   ] summary = ERROR
2018-11-20 02:20:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564747  13 58442  8405941 A  ERRORERROR'
2018-11-20 02:20:06 [DEBUG   ] Got command RPT from MCS: ref #564749
2018-11-20 02:20:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:20:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564749  98 58442  8406133 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:20:06 [DEBUG   ] Got command RPT from MCS: ref #564750
2018-11-20 02:20:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:20:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564750  12 58442  8406341 A  ERRORB£ff'
2018-11-20 02:20:10 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 02:20:30 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 02:20:46 [DEBUG   ] Got command RPT from MCS: ref #564755
2018-11-20 02:20:46 [DEBUG   ] summary = ERROR
2018-11-20 02:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564755  13 58442  8446577 A  ERRORERROR'
2018-11-20 02:20:46 [DEBUG   ] Got command RPT from MCS: ref #564756
2018-11-20 02:20:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564756  98 58442  8446578 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:20:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:20:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:20:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:20:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:20:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:20:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 02:20:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:20:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:20:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:20:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:20:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:20:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:20:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:20:49 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:20:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:20:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:20:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:20:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:20:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:20:49 [DEBUG   ] Finished updating board flags in 1.761 seconds
2018-11-20 02:20:50 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 02:21:10 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:21:30 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 02:21:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:21:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365380156000000
2018-11-20 02:21:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:21:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:21:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:21:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:21:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365380156000000
2018-11-20 02:21:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:21:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:21:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:21:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:21:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:21:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:21:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:21:49 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:21:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:21:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:21:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:21:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:21:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:21:49 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:21:49 [DEBUG   ] Finished updating board flags in 1.637 seconds
2018-11-20 02:21:50 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 02:22:10 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 02:22:30 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:22:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:22:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365391916000000
2018-11-20 02:22:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:22:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:22:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:22:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:22:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365391916000000
2018-11-20 02:22:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 02:22:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-20 02:22:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:22:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:22:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:22:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:22:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:22:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:22:49 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:22:49 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:22:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:22:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:22:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:22:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:22:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:22:49 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:22:49 [DEBUG   ] Finished updating board flags in 1.523 seconds
2018-11-20 02:22:50 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 02:23:10 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 02:23:30 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 02:23:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:23:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365403676000000
2018-11-20 02:23:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:23:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:23:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:23:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:23:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365403676000000
2018-11-20 02:23:48 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:23:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 02:23:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:23:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:23:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:23:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:23:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:23:49 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:23:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:23:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:23:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:23:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:23:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:23:49 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:23:49 [DEBUG   ] Finished updating board flags in 1.439 seconds
2018-11-20 02:23:50 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 02:24:10 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 02:24:30 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 02:24:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:24:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365415436000000
2018-11-20 02:24:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:24:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:24:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:24:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:24:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365415436000000
2018-11-20 02:24:48 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 02:24:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:24:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:24:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365415632000000
2018-11-20 02:24:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:24:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:24:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:24:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:24:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365415632000000
2018-11-20 02:24:49 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:24:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:24:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:24:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:24:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:24:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:24:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:24:49 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 02:24:49 [DEBUG   ] Finished updating board flags in 1.491 seconds
2018-11-20 02:24:50 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 02:25:05 [DEBUG   ] Got command RPT from MCS: ref #564776
2018-11-20 02:25:05 [DEBUG   ] summary = ERROR
2018-11-20 02:25:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564776  13 58442  8705617 A  ERRORERROR'
2018-11-20 02:25:05 [DEBUG   ] Got command RPT from MCS: ref #564777
2018-11-20 02:25:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:25:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564777  79 58442  8705821 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:25:06 [DEBUG   ] Got command RPT from MCS: ref #564778
2018-11-20 02:25:06 [DEBUG   ] summary = ERROR
2018-11-20 02:25:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564778  13 58442  8706013 A  ERRORERROR'
2018-11-20 02:25:06 [DEBUG   ] Got command RPT from MCS: ref #564779
2018-11-20 02:25:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:25:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564779  79 58442  8706217 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:25:06 [DEBUG   ] Got command RPT from MCS: ref #564780
2018-11-20 02:25:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:25:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564780  12 58442  8706421 A  ERRORB¤ff'
2018-11-20 02:25:10 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:25:30 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 02:25:46 [DEBUG   ] Got command RPT from MCS: ref #564785
2018-11-20 02:25:46 [DEBUG   ] summary = ERROR
2018-11-20 02:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564785  13 58442  8746577 A  ERRORERROR'
2018-11-20 02:25:46 [DEBUG   ] Got command RPT from MCS: ref #564786
2018-11-20 02:25:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564786  79 58442  8746578 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:25:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:25:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365427196000000
2018-11-20 02:25:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:25:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:25:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:25:48 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:25:48 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365427196000000
2018-11-20 02:25:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 02:25:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:25:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:25:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365427392000000
2018-11-20 02:25:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:25:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:25:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:25:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:25:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365427392000000
2018-11-20 02:25:49 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:25:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:25:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:25:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:25:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:25:49 [WARNING ] (13, 1, 'bot')
2018-11-20 02:25:49 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:25:49 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 02:25:49 [DEBUG   ] Finished updating board flags in 1.562 seconds
2018-11-20 02:25:50 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 02:26:10 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:26:30 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:26:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:26:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:26:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:26:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:26:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:26:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:26:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:26:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:26:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:26:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:26:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:26:50 [DEBUG   ] Finished updating ant stats in 0.648 seconds
2018-11-20 02:26:50 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:26:50 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:26:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:26:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:26:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:26:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:26:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:26:50 [DEBUG   ] Finished updating board flags in 2.168 seconds
2018-11-20 02:27:10 [DEBUG   ] Finished updating ant stats in 0.559 seconds
2018-11-20 02:27:30 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 02:27:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:27:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:27:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:27:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:27:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:27:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 02:27:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 02:27:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:27:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:27:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:27:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:27:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:27:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:27:50 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:27:50 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:27:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:27:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:27:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:27:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:27:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:27:50 [DEBUG   ] Finished updating board flags in 1.980 seconds
2018-11-20 02:27:51 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 02:28:11 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 02:28:31 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:28:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:28:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:28:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:28:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:28:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:28:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:28:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:28:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:28:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:28:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:28:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:28:50 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:28:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:28:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:28:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:28:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:28:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:28:50 [DEBUG   ] Finished updating board flags in 1.828 seconds
2018-11-20 02:28:51 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:29:11 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 02:29:31 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:29:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:29:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:29:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:29:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:29:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:29:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 02:29:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:29:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:29:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:29:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:29:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:29:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:29:50 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:29:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:29:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:29:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:29:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:29:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:29:50 [DEBUG   ] Finished updating board flags in 1.881 seconds
2018-11-20 02:29:51 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:30:05 [DEBUG   ] Got command RPT from MCS: ref #564808
2018-11-20 02:30:05 [DEBUG   ] summary = ERROR
2018-11-20 02:30:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564808  13 58442  9005701 A  ERRORERROR'
2018-11-20 02:30:05 [DEBUG   ] Got command RPT from MCS: ref #564809
2018-11-20 02:30:05 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:30:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564809  98 58442  9005897 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:30:06 [DEBUG   ] Got command RPT from MCS: ref #564810
2018-11-20 02:30:06 [DEBUG   ] summary = ERROR
2018-11-20 02:30:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564810  13 58442  9006101 A  ERRORERROR'
2018-11-20 02:30:06 [DEBUG   ] Got command RPT from MCS: ref #564812
2018-11-20 02:30:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:30:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564812  98 58442  9006305 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:30:06 [DEBUG   ] Got command RPT from MCS: ref #564813
2018-11-20 02:30:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:30:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564813  12 58442  9006497 A  ERRORB¤ff'
2018-11-20 02:30:11 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:30:31 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 02:30:46 [DEBUG   ] Got command RPT from MCS: ref #564818
2018-11-20 02:30:46 [DEBUG   ] summary = ERROR
2018-11-20 02:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564818  13 58442  9046569 A  ERRORERROR'
2018-11-20 02:30:46 [DEBUG   ] Got command RPT from MCS: ref #564819
2018-11-20 02:30:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564819  98 58442  9046570 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:30:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:30:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:30:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:30:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:30:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:30:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:30:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:30:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:30:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:30:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:30:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:30:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:30:50 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:30:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:30:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:30:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:30:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:30:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:30:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:30:50 [DEBUG   ] Finished updating board flags in 1.691 seconds
2018-11-20 02:30:51 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:31:11 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:31:31 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:31:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:31:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365497952000000
2018-11-20 02:31:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:31:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:31:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:31:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:31:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365497952000000
2018-11-20 02:31:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:31:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:31:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:31:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:31:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:31:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:31:50 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:31:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:31:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:31:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:31:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:31:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:31:50 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:31:50 [DEBUG   ] Finished updating board flags in 1.653 seconds
2018-11-20 02:31:51 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 02:32:11 [DEBUG   ] Finished updating ant stats in 1.191 seconds
2018-11-20 02:32:31 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 02:32:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:32:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365509712000000
2018-11-20 02:32:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:32:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:32:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:32:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:32:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365509712000000
2018-11-20 02:32:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:32:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:32:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:32:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:32:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:32:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:32:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:32:50 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:32:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:32:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:32:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:32:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:32:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:32:50 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:32:50 [DEBUG   ] Finished updating board flags in 1.547 seconds
2018-11-20 02:32:51 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:33:11 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 02:33:31 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 02:33:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:33:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365521472000000
2018-11-20 02:33:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:33:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:33:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:33:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:33:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365521472000000
2018-11-20 02:33:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 02:33:49 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-20 02:33:49 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:33:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:33:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:33:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:33:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:33:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:33:50 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:33:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:33:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:33:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:33:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:33:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:33:50 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:33:50 [DEBUG   ] Finished updating board flags in 1.458 seconds
2018-11-20 02:33:51 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 02:34:11 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:34:31 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 02:34:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:34:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365533232000000
2018-11-20 02:34:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:34:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:34:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:34:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:34:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365533232000000
2018-11-20 02:34:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:34:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:34:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:34:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365533428000000
2018-11-20 02:34:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:34:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:34:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:34:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:34:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365533428000000
2018-11-20 02:34:50 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:34:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:34:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:34:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:34:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:34:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:34:50 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 02:34:50 [DEBUG   ] Finished updating board flags in 1.455 seconds
2018-11-20 02:34:51 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 02:35:05 [DEBUG   ] Got command RPT from MCS: ref #564843
2018-11-20 02:35:05 [DEBUG   ] summary = ERROR
2018-11-20 02:35:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564843  13 58442  9305781 A  ERRORERROR'
2018-11-20 02:35:05 [DEBUG   ] Got command RPT from MCS: ref #564844
2018-11-20 02:35:05 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:35:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564844  79 58442  9305985 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:35:06 [DEBUG   ] Got command RPT from MCS: ref #564845
2018-11-20 02:35:06 [DEBUG   ] summary = ERROR
2018-11-20 02:35:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564845  13 58442  9306177 A  ERRORERROR'
2018-11-20 02:35:06 [DEBUG   ] Got command RPT from MCS: ref #564846
2018-11-20 02:35:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:35:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564846  79 58442  9306381 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:35:06 [DEBUG   ] Got command RPT from MCS: ref #564847
2018-11-20 02:35:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:35:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564847  12 58442  9306585 A  ERRORB£ff'
2018-11-20 02:35:11 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:35:31 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:35:46 [DEBUG   ] Got command RPT from MCS: ref #564848
2018-11-20 02:35:46 [DEBUG   ] summary = ERROR
2018-11-20 02:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564848  13 58442  9346573 A  ERRORERROR'
2018-11-20 02:35:46 [DEBUG   ] Got command RPT from MCS: ref #564849
2018-11-20 02:35:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564849  79 58442  9346574 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:35:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:35:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365544992000000
2018-11-20 02:35:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:35:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:35:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:35:49 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:35:49 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365544992000000
2018-11-20 02:35:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-20 02:35:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:35:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:35:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365545188000000
2018-11-20 02:35:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:35:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:35:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:35:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:35:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365545188000000
2018-11-20 02:35:50 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:35:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:35:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:35:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:35:50 [WARNING ] (13, 1, 'bot')
2018-11-20 02:35:50 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:35:50 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 02:35:50 [DEBUG   ] Finished updating board flags in 1.506 seconds
2018-11-20 02:35:51 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 02:36:11 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 02:36:31 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 02:36:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:36:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:36:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:36:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:36:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:36:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 02:36:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:36:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 02:36:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:36:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:36:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:36:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:36:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:36:51 [DEBUG   ] Finished updating ant stats in 0.646 seconds
2018-11-20 02:36:51 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:36:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:36:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 02:36:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:36:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:36:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:36:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:36:51 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:36:51 [DEBUG   ] Finished updating board flags in 2.227 seconds
2018-11-20 02:37:11 [DEBUG   ] Finished updating ant stats in 0.560 seconds
2018-11-20 02:37:32 [DEBUG   ] Finished updating ant stats in 1.394 seconds
2018-11-20 02:37:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:37:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:37:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:37:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:37:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:37:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:37:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:37:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:37:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:37:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:37:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:37:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:37:51 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:37:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:37:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:37:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:37:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:37:51 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:37:51 [DEBUG   ] Finished updating board flags in 2.156 seconds
2018-11-20 02:37:52 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 02:38:12 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:38:32 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:38:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:38:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:38:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:38:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:38:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:38:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 02:38:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:38:50 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 02:38:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:38:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:38:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:38:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:38:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:38:52 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:38:52 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:38:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:38:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:38:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:38:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:38:52 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:38:52 [DEBUG   ] Finished updating board flags in 3.123 seconds
2018-11-20 02:39:12 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:39:32 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:39:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:39:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:39:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:39:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:39:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:39:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 02:39:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:39:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:39:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:39:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:39:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:39:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:39:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 02:39:51 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:39:51 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:39:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:39:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:39:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:39:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:39:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:39:51 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:39:51 [DEBUG   ] Finished updating board flags in 1.908 seconds
2018-11-20 02:39:52 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:40:05 [DEBUG   ] Got command RPT from MCS: ref #564871
2018-11-20 02:40:05 [DEBUG   ] summary = ERROR
2018-11-20 02:40:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564871  13 58442  9605857 A  ERRORERROR'
2018-11-20 02:40:06 [DEBUG   ] Got command RPT from MCS: ref #564872
2018-11-20 02:40:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:40:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564872  98 58442  9606062 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:40:06 [DEBUG   ] Got command RPT from MCS: ref #564873
2018-11-20 02:40:06 [DEBUG   ] summary = ERROR
2018-11-20 02:40:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564873  13 58442  9606265 A  ERRORERROR'
2018-11-20 02:40:06 [DEBUG   ] Got command RPT from MCS: ref #564875
2018-11-20 02:40:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:40:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564875  98 58442  9606457 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:40:06 [DEBUG   ] Got command RPT from MCS: ref #564876
2018-11-20 02:40:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:40:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564876  12 58442  9606661 A  ERRORB¤ff'
2018-11-20 02:40:12 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:40:32 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:40:46 [DEBUG   ] Got command RPT from MCS: ref #564885
2018-11-20 02:40:46 [DEBUG   ] summary = ERROR
2018-11-20 02:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564885  13 58442  9646573 A  ERRORERROR'
2018-11-20 02:40:46 [DEBUG   ] Got command RPT from MCS: ref #564886
2018-11-20 02:40:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564886  98 58442  9646574 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:40:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:40:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:40:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:40:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:40:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:40:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:40:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:40:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:40:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:40:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:40:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:40:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:40:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 02:40:51 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:40:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:40:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:40:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:40:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:40:51 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:40:51 [DEBUG   ] Finished updating board flags in 1.877 seconds
2018-11-20 02:40:52 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:41:12 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:41:32 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:41:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:41:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:41:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:41:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:41:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:41:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:41:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 02:41:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:41:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:41:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:41:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:41:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:41:51 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:41:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:41:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:41:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:41:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:41:51 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:41:51 [DEBUG   ] Finished updating board flags in 1.821 seconds
2018-11-20 02:41:52 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:42:12 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:42:32 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:42:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:42:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:42:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:42:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:42:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:42:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:42:50 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:42:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:42:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:42:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:42:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:42:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:42:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 02:42:51 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:42:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:42:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:42:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:42:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:42:51 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:42:51 [DEBUG   ] Finished updating board flags in 1.698 seconds
2018-11-20 02:42:52 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:43:12 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:43:32 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 02:43:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:43:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365639268000000
2018-11-20 02:43:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:43:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:43:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:43:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:43:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365639268000000
2018-11-20 02:43:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:43:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:43:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:43:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:43:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:43:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:43:51 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:43:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:43:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:43:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:43:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:43:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:43:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:43:51 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:43:51 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:43:51 [DEBUG   ] Finished updating board flags in 1.611 seconds
2018-11-20 02:43:52 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:44:12 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:44:32 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 02:44:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:44:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365651028000000
2018-11-20 02:44:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:44:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:44:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:44:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:44:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365651028000000
2018-11-20 02:44:50 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:44:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:44:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:44:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:44:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:44:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:44:51 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:44:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:44:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:44:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:44:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:44:51 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:44:51 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:44:51 [DEBUG   ] Finished updating board flags in 1.579 seconds
2018-11-20 02:44:52 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 02:45:05 [DEBUG   ] Got command RPT from MCS: ref #564906
2018-11-20 02:45:05 [DEBUG   ] summary = ERROR
2018-11-20 02:45:05 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564906  13 58442  9905937 A  ERRORERROR'
2018-11-20 02:45:06 [DEBUG   ] Got command RPT from MCS: ref #564907
2018-11-20 02:45:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 02:45:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564907  71 58442  9906141 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 02:45:06 [DEBUG   ] Got command RPT from MCS: ref #564908
2018-11-20 02:45:06 [DEBUG   ] summary = ERROR
2018-11-20 02:45:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564908  13 58442  9906345 A  ERRORERROR'
2018-11-20 02:45:06 [DEBUG   ] Got command RPT from MCS: ref #564909
2018-11-20 02:45:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 02:45:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564909  71 58442  9906537 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 02:45:06 [DEBUG   ] Got command RPT from MCS: ref #564910
2018-11-20 02:45:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:45:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564910  12 58442  9906742 A  ERRORB£ff'
2018-11-20 02:45:12 [DEBUG   ] Finished updating ant stats in 0.991 seconds
2018-11-20 02:45:32 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 02:45:46 [DEBUG   ] Got command RPT from MCS: ref #564911
2018-11-20 02:45:46 [DEBUG   ] summary = ERROR
2018-11-20 02:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564911  13 58442  9946573 A  ERRORERROR'
2018-11-20 02:45:46 [DEBUG   ] Got command RPT from MCS: ref #564912
2018-11-20 02:45:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 02:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564912  71 58442  9946574 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 02:45:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:45:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365662788000000
2018-11-20 02:45:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:45:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:45:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:45:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:45:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365662788000000
2018-11-20 02:45:50 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 02:45:51 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:45:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:45:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:45:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365662984000000
2018-11-20 02:45:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:45:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:45:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:45:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:45:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365662984000000
2018-11-20 02:45:51 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:45:52 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 02:45:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:45:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:45:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:45:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:45:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:45:52 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:45:52 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 02:45:52 [DEBUG   ] Finished updating board flags in 2.252 seconds
2018-11-20 02:46:12 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 02:46:32 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 02:46:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:46:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365674548000000
2018-11-20 02:46:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:46:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:46:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:46:50 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:46:50 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365674548000000
2018-11-20 02:46:51 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:46:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:46:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:46:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365674744000000
2018-11-20 02:46:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:46:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:46:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:46:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:46:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365674744000000
2018-11-20 02:46:51 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:46:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:46:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:46:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:46:51 [WARNING ] (13, 1, 'bot')
2018-11-20 02:46:51 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:46:51 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 02:46:51 [DEBUG   ] Finished updating board flags in 1.528 seconds
2018-11-20 02:46:52 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:47:12 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 02:47:32 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 02:47:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:47:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:47:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:47:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:47:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:47:51 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:47:52 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 02:47:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:47:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365686700000000
2018-11-20 02:47:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:47:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:47:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:47:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:47:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365686700000000
2018-11-20 02:47:52 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:47:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:47:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:47:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:47:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:47:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:47:53 [CRITICAL] DRX time not updating correctly on beam(s) #2, #1
2018-11-20 02:47:53 [DEBUG   ] Finished updating board flags in 2.554 seconds
2018-11-20 02:48:12 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 02:48:32 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 02:48:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:48:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:48:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:48:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:48:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:48:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 02:48:51 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:48:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:48:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:48:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:48:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:48:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:48:52 [DEBUG   ] Finished updating ant stats in 0.652 seconds
2018-11-20 02:48:52 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:48:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:48:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:48:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:48:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:48:52 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:48:52 [DEBUG   ] Finished updating board flags in 2.158 seconds
2018-11-20 02:49:12 [DEBUG   ] Finished updating ant stats in 0.550 seconds
2018-11-20 02:49:32 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-20 02:49:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:49:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:49:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:49:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:49:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:49:51 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:49:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:49:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:49:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:49:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:49:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:49:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:49:52 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:49:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:49:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:49:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:49:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:49:52 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:49:52 [DEBUG   ] Finished updating board flags in 1.980 seconds
2018-11-20 02:49:53 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 02:50:06 [DEBUG   ] Got command RPT from MCS: ref #564938
2018-11-20 02:50:06 [DEBUG   ] summary = ERROR
2018-11-20 02:50:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564938  13 58442 10206021 A  ERRORERROR'
2018-11-20 02:50:06 [DEBUG   ] Got command RPT from MCS: ref #564939
2018-11-20 02:50:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:50:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564939  98 58442 10206225 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:50:06 [DEBUG   ] Got command RPT from MCS: ref #564940
2018-11-20 02:50:06 [DEBUG   ] summary = ERROR
2018-11-20 02:50:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564940  13 58442 10206417 A  ERRORERROR'
2018-11-20 02:50:06 [DEBUG   ] Got command RPT from MCS: ref #564942
2018-11-20 02:50:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:50:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564942  98 58442 10206621 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:50:06 [DEBUG   ] Got command RPT from MCS: ref #564944
2018-11-20 02:50:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:50:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564944  12 58442 10206825 A  ERRORB¤ff'
2018-11-20 02:50:13 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 02:50:33 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:50:46 [DEBUG   ] Got command RPT from MCS: ref #564948
2018-11-20 02:50:46 [DEBUG   ] summary = ERROR
2018-11-20 02:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564948  13 58442 10246581 A  ERRORERROR'
2018-11-20 02:50:46 [DEBUG   ] Got command RPT from MCS: ref #564949
2018-11-20 02:50:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 02:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564949  98 58442 10246582 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 02:50:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:50:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:50:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:50:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:50:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:50:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 02:50:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 02:50:51 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:50:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:50:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:50:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:50:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:50:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:50:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 02:50:52 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:50:52 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:50:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:50:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:50:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:50:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:50:52 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:50:52 [DEBUG   ] Finished updating board flags in 1.761 seconds
2018-11-20 02:50:53 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:51:13 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:51:33 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:51:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:51:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365733544000000
2018-11-20 02:51:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:51:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:51:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:51:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:51:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365733544000000
2018-11-20 02:51:51 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 02:51:51 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:51:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:51:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:51:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:51:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:51:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:51:52 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:51:52 [WARNING ] MonitorFlags: VCCINT 0.976 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:51:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:51:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:51:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:51:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:51:52 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:51:52 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:51:52 [DEBUG   ] Finished updating board flags in 1.612 seconds
2018-11-20 02:51:53 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:52:13 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:52:33 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 02:52:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:52:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365745304000000
2018-11-20 02:52:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:52:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:52:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:52:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:52:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365745304000000
2018-11-20 02:52:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 02:52:51 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 02:52:51 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:52:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:52:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:52:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:52:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:52:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:52:52 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:52:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:52:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:52:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:52:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:52:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:52:52 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:52:52 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:52:52 [DEBUG   ] Finished updating board flags in 1.626 seconds
2018-11-20 02:52:53 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:53:13 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:53:33 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:53:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:53:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365757064000000
2018-11-20 02:53:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:53:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:53:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:53:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:53:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365757064000000
2018-11-20 02:53:51 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 02:53:51 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:53:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:53:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:53:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:53:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:53:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:53:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 02:53:52 [WARNING ] MonitorFlags: Temperature 82.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:53:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:53:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:53:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:53:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:53:52 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:53:52 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 02:53:52 [DEBUG   ] Finished updating board flags in 1.585 seconds
2018-11-20 02:53:53 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 02:54:13 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 02:54:33 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:54:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:54:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365768824000000
2018-11-20 02:54:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:54:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:54:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:54:51 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:54:51 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365768824000000
2018-11-20 02:54:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:54:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:54:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365769020000000
2018-11-20 02:54:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:54:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:54:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:54:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:54:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365769020000000
2018-11-20 02:54:52 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:54:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:54:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:54:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:54:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:54:52 [WARNING ] (13, 1, 'bot')
2018-11-20 02:54:52 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:54:52 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 02:54:52 [DEBUG   ] Finished updating board flags in 1.569 seconds
2018-11-20 02:54:53 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 02:55:06 [DEBUG   ] Got command RPT from MCS: ref #564969
2018-11-20 02:55:06 [DEBUG   ] summary = ERROR
2018-11-20 02:55:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564969  13 58442 10506097 A  ERRORERROR'
2018-11-20 02:55:06 [DEBUG   ] Got command RPT from MCS: ref #564970
2018-11-20 02:55:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:55:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564970  79 58442 10506301 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:55:06 [DEBUG   ] Got command RPT from MCS: ref #564971
2018-11-20 02:55:06 [DEBUG   ] summary = ERROR
2018-11-20 02:55:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564971  13 58442 10506505 A  ERRORERROR'
2018-11-20 02:55:06 [DEBUG   ] Got command RPT from MCS: ref #564972
2018-11-20 02:55:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:55:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564972  79 58442 10506697 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:55:06 [DEBUG   ] Got command RPT from MCS: ref #564973
2018-11-20 02:55:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 02:55:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564973  12 58442 10506901 A  ERRORB¤ff'
2018-11-20 02:55:13 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 02:55:33 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 02:55:46 [DEBUG   ] Got command RPT from MCS: ref #564978
2018-11-20 02:55:46 [DEBUG   ] summary = ERROR
2018-11-20 02:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564978  13 58442 10546577 A  ERRORERROR'
2018-11-20 02:55:46 [DEBUG   ] Got command RPT from MCS: ref #564979
2018-11-20 02:55:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 02:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   564979  79 58442 10546578 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 02:55:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:55:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:55:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:55:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:55:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:55:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 02:55:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:55:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:55:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:55:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:55:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:55:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:55:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 02:55:53 [DEBUG   ] Finished updating ant stats in 1.054 seconds
2018-11-20 02:55:53 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:55:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 02:55:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:55:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:55:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:55:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:55:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:55:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:55:53 [DEBUG   ] Finished updating board flags in 2.282 seconds
2018-11-20 02:56:13 [DEBUG   ] Finished updating ant stats in 0.949 seconds
2018-11-20 02:56:33 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 02:56:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:56:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:56:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:56:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:56:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:56:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:56:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:56:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:56:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:56:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:56:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:56:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:56:53 [DEBUG   ] Finished updating ant stats in 0.853 seconds
2018-11-20 02:56:53 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:56:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:56:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:56:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:56:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:56:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:56:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:56:53 [DEBUG   ] Finished updating board flags in 2.159 seconds
2018-11-20 02:57:13 [DEBUG   ] Finished updating ant stats in 0.749 seconds
2018-11-20 02:57:33 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 02:57:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:57:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:57:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:57:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:57:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:57:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:57:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 02:57:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:57:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:57:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:57:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:57:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:57:53 [DEBUG   ] Finished updating ant stats in 0.643 seconds
2018-11-20 02:57:53 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 02:57:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:57:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 02:57:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:57:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:57:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:57:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:57:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:57:53 [DEBUG   ] Finished updating board flags in 1.985 seconds
2018-11-20 02:58:13 [DEBUG   ] Finished updating ant stats in 0.558 seconds
2018-11-20 02:58:33 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-20 02:58:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:58:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:58:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:58:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:58:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:58:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 02:58:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:58:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:58:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:58:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:58:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:58:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:58:53 [DEBUG   ] Finished updating ant stats in 0.448 seconds
2018-11-20 02:58:53 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:58:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:58:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:58:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:58:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:58:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:58:53 [DEBUG   ] Finished updating board flags in 1.810 seconds
2018-11-20 02:59:14 [DEBUG   ] Finished updating ant stats in 1.358 seconds
2018-11-20 02:59:34 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 02:59:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 02:59:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 02:59:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 02:59:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 02:59:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 02:59:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 02:59:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 02:59:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 02:59:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 02:59:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 02:59:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 02:59:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 02:59:53 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 02:59:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 02:59:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:59:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:59:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:59:53 [WARNING ] (13, 1, 'bot')
2018-11-20 02:59:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 02:59:53 [DEBUG   ] Finished updating board flags in 1.779 seconds
2018-11-20 02:59:54 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 03:00:06 [DEBUG   ] Got command RPT from MCS: ref #565001
2018-11-20 03:00:06 [DEBUG   ] summary = ERROR
2018-11-20 03:00:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565001  13 58442 10806177 A  ERRORERROR'
2018-11-20 03:00:06 [DEBUG   ] Got command RPT from MCS: ref #565002
2018-11-20 03:00:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:00:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565002  98 58442 10806381 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:00:06 [DEBUG   ] Got command RPT from MCS: ref #565003
2018-11-20 03:00:06 [DEBUG   ] summary = ERROR
2018-11-20 03:00:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565003  13 58442 10806585 A  ERRORERROR'
2018-11-20 03:00:06 [DEBUG   ] Got command RPT from MCS: ref #565005
2018-11-20 03:00:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:00:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565005  98 58442 10806789 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:00:06 [DEBUG   ] Got command RPT from MCS: ref #565007
2018-11-20 03:00:06 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:00:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565007  12 58442 10806981 A  ERRORB£ff'
2018-11-20 03:00:14 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 03:00:34 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 03:00:46 [DEBUG   ] Got command RPT from MCS: ref #565011
2018-11-20 03:00:46 [DEBUG   ] summary = ERROR
2018-11-20 03:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565011  13 58442 10846577 A  ERRORERROR'
2018-11-20 03:00:46 [DEBUG   ] Got command RPT from MCS: ref #565012
2018-11-20 03:00:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565012  98 58442 10846578 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:00:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:00:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365839580000000
2018-11-20 03:00:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:00:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:00:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:00:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:00:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365839580000000
2018-11-20 03:00:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:00:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:00:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:00:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:00:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:00:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:00:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 03:00:53 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:00:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:00:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:00:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:00:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:00:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:00:53 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:00:53 [DEBUG   ] Finished updating board flags in 1.614 seconds
2018-11-20 03:00:54 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:01:14 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:01:34 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:01:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:01:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365851340000000
2018-11-20 03:01:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:01:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:01:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:01:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:01:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365851340000000
2018-11-20 03:01:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:01:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:01:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:01:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:01:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:01:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:01:53 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:01:53 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:01:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:01:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:01:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:01:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:01:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:01:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:01:53 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:01:53 [DEBUG   ] Finished updating board flags in 1.584 seconds
2018-11-20 03:01:54 [DEBUG   ] Finished updating ant stats in 0.993 seconds
2018-11-20 03:02:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:02:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:02:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:02:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365863100000000
2018-11-20 03:02:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:02:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:02:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:02:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:02:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365863100000000
2018-11-20 03:02:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 03:02:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 03:02:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 03:02:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 03:02:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:02:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:02:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:02:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:02:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:02:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:02:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:02:53 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:02:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:02:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:02:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:02:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:02:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:02:53 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:02:53 [DEBUG   ] Finished updating board flags in 1.419 seconds
2018-11-20 03:02:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 03:03:14 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 03:03:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:03:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:03:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365874860000000
2018-11-20 03:03:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:03:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:03:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:03:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:03:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365874860000000
2018-11-20 03:03:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:03:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:03:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:03:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:03:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:03:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:03:53 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:03:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:03:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:03:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:03:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:03:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:03:53 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:03:53 [DEBUG   ] Finished updating board flags in 1.419 seconds
2018-11-20 03:03:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 03:04:14 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 03:04:34 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 03:04:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:04:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365886620000000
2018-11-20 03:04:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:04:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:04:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:04:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:04:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365886620000000
2018-11-20 03:04:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 03:04:52 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:04:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:04:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365886816000000
2018-11-20 03:04:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:04:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:04:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:04:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:04:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365886816000000
2018-11-20 03:04:53 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:04:53 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:04:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:04:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:04:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:04:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:04:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:04:53 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:04:53 [DEBUG   ] Finished updating board flags in 1.487 seconds
2018-11-20 03:04:54 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 03:05:06 [DEBUG   ] Got command RPT from MCS: ref #565036
2018-11-20 03:05:06 [DEBUG   ] summary = ERROR
2018-11-20 03:05:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565036  13 58442 11106276 A  ERRORERROR'
2018-11-20 03:05:06 [DEBUG   ] Got command RPT from MCS: ref #565037
2018-11-20 03:05:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:05:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565037  79 58442 11106466 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:05:06 [DEBUG   ] Got command RPT from MCS: ref #565038
2018-11-20 03:05:06 [DEBUG   ] summary = ERROR
2018-11-20 03:05:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565038  13 58442 11106657 A  ERRORERROR'
2018-11-20 03:05:06 [DEBUG   ] Got command RPT from MCS: ref #565039
2018-11-20 03:05:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:05:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565039  79 58442 11106861 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:05:07 [DEBUG   ] Got command RPT from MCS: ref #565040
2018-11-20 03:05:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:05:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565040  12 58442 11107065 A  ERRORB¤ff'
2018-11-20 03:05:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:05:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 03:05:46 [DEBUG   ] Got command RPT from MCS: ref #565041
2018-11-20 03:05:46 [DEBUG   ] summary = ERROR
2018-11-20 03:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565041  13 58442 11146585 A  ERRORERROR'
2018-11-20 03:05:46 [DEBUG   ] Got command RPT from MCS: ref #565042
2018-11-20 03:05:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565042  79 58442 11146586 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:05:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:05:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365898380000000
2018-11-20 03:05:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:05:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:05:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:05:52 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:05:52 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365898380000000
2018-11-20 03:05:52 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:05:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:05:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:05:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302365898576000000
2018-11-20 03:05:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:05:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:05:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:05:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:05:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302365898576000000
2018-11-20 03:05:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 03:05:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 03:05:53 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:05:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:05:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:05:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:05:53 [WARNING ] (13, 1, 'bot')
2018-11-20 03:05:53 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:05:53 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:05:53 [DEBUG   ] Finished updating board flags in 1.497 seconds
2018-11-20 03:05:54 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 03:06:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:06:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 03:06:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:06:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:06:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:06:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:06:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:06:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 03:06:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:06:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:06:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:06:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:06:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:06:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:06:54 [DEBUG   ] Finished updating ant stats in 0.847 seconds
2018-11-20 03:06:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 03:06:54 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:06:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:06:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:06:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:06:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:06:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:06:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:06:54 [DEBUG   ] Finished updating board flags in 2.264 seconds
2018-11-20 03:07:14 [DEBUG   ] Finished updating ant stats in 0.757 seconds
2018-11-20 03:07:34 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 03:07:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:07:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:07:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:07:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:07:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:07:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:07:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:07:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:07:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:07:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:07:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:07:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:07:54 [DEBUG   ] Finished updating ant stats in 0.645 seconds
2018-11-20 03:07:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 03:07:54 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:07:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:07:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:07:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:07:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:07:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:07:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:07:54 [DEBUG   ] Finished updating board flags in 2.156 seconds
2018-11-20 03:08:14 [DEBUG   ] Finished updating ant stats in 0.559 seconds
2018-11-20 03:08:35 [DEBUG   ] Finished updating ant stats in 1.402 seconds
2018-11-20 03:08:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:08:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:08:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:08:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:08:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:08:53 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 03:08:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 03:08:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:08:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:08:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:08:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:08:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:08:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:08:54 [DEBUG   ] Finished updating ant stats in 0.445 seconds
2018-11-20 03:08:54 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:08:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:08:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:08:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:08:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:08:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:08:54 [DEBUG   ] Finished updating board flags in 1.989 seconds
2018-11-20 03:09:15 [DEBUG   ] Finished updating ant stats in 1.358 seconds
2018-11-20 03:09:35 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 03:09:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:09:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:09:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:09:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:09:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:09:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:09:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:09:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:09:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:09:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:09:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:09:54 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:09:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:09:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:09:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:09:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:09:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:09:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:09:54 [DEBUG   ] Finished updating board flags in 1.846 seconds
2018-11-20 03:09:55 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 03:10:06 [DEBUG   ] Got command RPT from MCS: ref #565064
2018-11-20 03:10:06 [DEBUG   ] summary = ERROR
2018-11-20 03:10:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565064  13 58442 11406341 A  ERRORERROR'
2018-11-20 03:10:06 [DEBUG   ] Got command RPT from MCS: ref #565065
2018-11-20 03:10:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:10:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565065  98 58442 11406545 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:10:06 [DEBUG   ] Got command RPT from MCS: ref #565066
2018-11-20 03:10:06 [DEBUG   ] summary = ERROR
2018-11-20 03:10:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565066  13 58442 11406737 A  ERRORERROR'
2018-11-20 03:10:06 [DEBUG   ] Got command RPT from MCS: ref #565068
2018-11-20 03:10:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:10:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565068  98 58442 11406942 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:10:07 [DEBUG   ] Got command RPT from MCS: ref #565070
2018-11-20 03:10:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:10:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565070  12 58442 11407145 A  ERRORB£ff'
2018-11-20 03:10:15 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 03:10:35 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:10:46 [DEBUG   ] Got command RPT from MCS: ref #565078
2018-11-20 03:10:46 [DEBUG   ] summary = ERROR
2018-11-20 03:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565078  13 58442 11446589 A  ERRORERROR'
2018-11-20 03:10:46 [DEBUG   ] Got command RPT from MCS: ref #565079
2018-11-20 03:10:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565079  98 58442 11446590 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:10:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:10:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:10:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:10:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:10:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:10:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:10:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:10:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:10:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:10:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:10:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:10:54 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:10:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:10:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:10:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:10:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:10:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:10:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:10:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:10:54 [DEBUG   ] Finished updating board flags in 1.773 seconds
2018-11-20 03:10:55 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:11:15 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:11:35 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:11:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:11:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365969136000000
2018-11-20 03:11:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:11:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:11:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:11:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:11:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365969136000000
2018-11-20 03:11:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:11:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:11:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 03:11:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:11:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:11:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:11:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:11:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:11:54 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:11:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:11:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:11:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:11:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:11:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:11:54 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:11:54 [DEBUG   ] Finished updating board flags in 1.621 seconds
2018-11-20 03:11:55 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:12:15 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:12:35 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:12:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:12:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365980896000000
2018-11-20 03:12:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:12:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:12:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:12:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:12:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365980896000000
2018-11-20 03:12:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-20 03:12:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:12:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:12:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:12:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:12:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:12:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:12:54 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:12:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:12:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:12:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:12:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:12:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:12:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:12:54 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:12:54 [DEBUG   ] Finished updating board flags in 1.558 seconds
2018-11-20 03:12:55 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 03:13:15 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:13:35 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:13:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:13:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302365992656000000
2018-11-20 03:13:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:13:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:13:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:13:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:13:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302365992656000000
2018-11-20 03:13:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 03:13:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:13:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:13:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:13:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:13:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:13:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:13:54 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:13:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:13:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:13:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:13:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:13:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:13:54 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:13:54 [DEBUG   ] Finished updating board flags in 1.423 seconds
2018-11-20 03:13:55 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:14:15 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:14:35 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:14:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:14:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366004416000000
2018-11-20 03:14:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:14:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:14:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:14:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:14:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366004416000000
2018-11-20 03:14:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:14:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:14:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366004612000000
2018-11-20 03:14:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:14:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:14:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:14:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:14:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366004612000000
2018-11-20 03:14:54 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:14:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:14:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:14:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:14:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:14:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:14:54 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:14:54 [DEBUG   ] Finished updating board flags in 1.606 seconds
2018-11-20 03:14:55 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 03:15:06 [DEBUG   ] Got command RPT from MCS: ref #565099
2018-11-20 03:15:06 [DEBUG   ] summary = ERROR
2018-11-20 03:15:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565099  13 58442 11706421 A  ERRORERROR'
2018-11-20 03:15:06 [DEBUG   ] Got command RPT from MCS: ref #565100
2018-11-20 03:15:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:15:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565100  79 58442 11706625 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:15:06 [DEBUG   ] Got command RPT from MCS: ref #565101
2018-11-20 03:15:06 [DEBUG   ] summary = ERROR
2018-11-20 03:15:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565101  13 58442 11706829 A  ERRORERROR'
2018-11-20 03:15:07 [DEBUG   ] Got command RPT from MCS: ref #565102
2018-11-20 03:15:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:15:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565102  79 58442 11707021 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:15:07 [DEBUG   ] Got command RPT from MCS: ref #565103
2018-11-20 03:15:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:15:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565103  12 58442 11707225 A  ERRORB¤ff'
2018-11-20 03:15:15 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 03:15:35 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 03:15:46 [DEBUG   ] Got command RPT from MCS: ref #565104
2018-11-20 03:15:46 [DEBUG   ] summary = ERROR
2018-11-20 03:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565104  13 58442 11746581 A  ERRORERROR'
2018-11-20 03:15:46 [DEBUG   ] Got command RPT from MCS: ref #565105
2018-11-20 03:15:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565105  79 58442 11746582 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:15:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:15:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366016176000000
2018-11-20 03:15:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:15:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:15:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:15:53 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:15:53 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366016176000000
2018-11-20 03:15:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:15:53 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:15:53 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 03:15:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:15:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:15:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:15:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:15:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:15:54 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:15:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:15:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:15:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:15:54 [WARNING ] (13, 1, 'bot')
2018-11-20 03:15:54 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:15:54 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:15:54 [DEBUG   ] Finished updating board flags in 1.377 seconds
2018-11-20 03:15:55 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 03:16:15 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 03:16:35 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 03:16:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:16:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:16:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:16:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:16:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:16:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:16:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:16:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:16:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:16:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:16:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:16:55 [DEBUG   ] Finished updating ant stats in 0.647 seconds
2018-11-20 03:16:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:16:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:16:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:16:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:16:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:16:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:16:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:16:55 [DEBUG   ] Finished updating board flags in 2.223 seconds
2018-11-20 03:17:15 [DEBUG   ] Finished updating ant stats in 0.558 seconds
2018-11-20 03:17:35 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 03:17:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:17:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:17:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:17:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:17:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:17:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:17:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:17:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:17:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:17:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:17:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:17:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:17:55 [DEBUG   ] Finished updating ant stats in 0.444 seconds
2018-11-20 03:17:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:17:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:17:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:17:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:17:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:17:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:17:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:17:55 [DEBUG   ] Finished updating board flags in 2.193 seconds
2018-11-20 03:18:16 [DEBUG   ] Finished updating ant stats in 1.359 seconds
2018-11-20 03:18:36 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 03:18:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:18:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:18:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:18:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:18:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:18:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 03:18:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:18:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:18:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:18:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:18:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:18:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:18:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:18:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 03:18:55 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:18:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:18:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:18:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:18:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:18:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:18:55 [DEBUG   ] Finished updating board flags in 2.020 seconds
2018-11-20 03:18:56 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 03:19:16 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 03:19:36 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 03:19:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:19:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:19:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:19:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:19:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:19:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 03:19:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-20 03:19:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:19:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:19:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:19:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:19:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:19:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:19:55 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:19:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:19:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:19:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:19:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:19:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:19:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:19:55 [DEBUG   ] Finished updating board flags in 2.017 seconds
2018-11-20 03:19:56 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 03:20:06 [DEBUG   ] Got command RPT from MCS: ref #565131
2018-11-20 03:20:06 [DEBUG   ] summary = ERROR
2018-11-20 03:20:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565131  13 58442 12006505 A  ERRORERROR'
2018-11-20 03:20:06 [DEBUG   ] Got command RPT from MCS: ref #565132
2018-11-20 03:20:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:20:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565132  98 58442 12006709 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:20:06 [DEBUG   ] Got command RPT from MCS: ref #565133
2018-11-20 03:20:06 [DEBUG   ] summary = ERROR
2018-11-20 03:20:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565133  13 58442 12006901 A  ERRORERROR'
2018-11-20 03:20:07 [DEBUG   ] Got command RPT from MCS: ref #565135
2018-11-20 03:20:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:20:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565135  98 58442 12007105 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:20:07 [DEBUG   ] Got command RPT from MCS: ref #565136
2018-11-20 03:20:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:20:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565136  12 58442 12007310 A  ERRORB£ff'
2018-11-20 03:20:16 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 03:20:36 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 03:20:46 [DEBUG   ] Got command RPT from MCS: ref #565141
2018-11-20 03:20:46 [DEBUG   ] summary = ERROR
2018-11-20 03:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565141  13 58442 12046589 A  ERRORERROR'
2018-11-20 03:20:46 [DEBUG   ] Got command RPT from MCS: ref #565142
2018-11-20 03:20:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565142  98 58442 12046590 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:20:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:20:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:20:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:20:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:20:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:20:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:20:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:20:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:20:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:20:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:20:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:20:55 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:20:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:20:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:20:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:20:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:20:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:20:55 [DEBUG   ] Finished updating board flags in 1.970 seconds
2018-11-20 03:20:56 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 03:21:16 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 03:21:36 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:21:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:21:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366086932000000
2018-11-20 03:21:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:21:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:21:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:21:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:21:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366086932000000
2018-11-20 03:21:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 03:21:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:21:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:21:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:21:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:21:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:21:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:21:55 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:21:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:21:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:21:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:21:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:21:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:21:55 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:21:55 [DEBUG   ] Finished updating board flags in 1.801 seconds
2018-11-20 03:21:56 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:22:16 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:22:36 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:22:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:22:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:22:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:22:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:22:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:22:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 03:22:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:22:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:22:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:22:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:22:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:22:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:22:55 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:22:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:22:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:22:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:22:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:22:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:22:55 [DEBUG   ] Finished updating board flags in 1.816 seconds
2018-11-20 03:22:56 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:23:16 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:23:36 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:23:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:23:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:23:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:23:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:23:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:23:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:23:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:23:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:23:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:23:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:23:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:23:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:23:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:23:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:23:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:23:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:23:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:23:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:23:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:23:55 [DEBUG   ] Finished updating board flags in 1.816 seconds
2018-11-20 03:23:56 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:24:16 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:24:36 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:24:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:24:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:24:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:24:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:24:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:24:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-20 03:24:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:24:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:24:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:24:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:24:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:24:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:24:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:24:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:24:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:24:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:24:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:24:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:24:55 [DEBUG   ] Finished updating board flags in 1.817 seconds
2018-11-20 03:24:56 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:25:06 [DEBUG   ] Got command RPT from MCS: ref #565162
2018-11-20 03:25:06 [DEBUG   ] summary = ERROR
2018-11-20 03:25:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565162  13 58442 12306589 A  ERRORERROR'
2018-11-20 03:25:06 [DEBUG   ] Got command RPT from MCS: ref #565163
2018-11-20 03:25:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:25:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565163  98 58442 12306781 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:25:06 [DEBUG   ] Got command RPT from MCS: ref #565164
2018-11-20 03:25:06 [DEBUG   ] summary = ERROR
2018-11-20 03:25:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565164  13 58442 12306986 A  ERRORERROR'
2018-11-20 03:25:07 [DEBUG   ] Got command RPT from MCS: ref #565165
2018-11-20 03:25:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:25:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565165  98 58442 12307189 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:25:07 [DEBUG   ] Got command RPT from MCS: ref #565166
2018-11-20 03:25:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:25:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565166  12 58442 12307382 A  ERRORB¤ff'
2018-11-20 03:25:16 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:25:36 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 03:25:46 [DEBUG   ] Got command RPT from MCS: ref #565171
2018-11-20 03:25:46 [DEBUG   ] summary = ERROR
2018-11-20 03:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565171  13 58442 12346585 A  ERRORERROR'
2018-11-20 03:25:46 [DEBUG   ] Got command RPT from MCS: ref #565172
2018-11-20 03:25:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565172  98 58442 12346586 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:25:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:25:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:25:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:25:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:25:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:25:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 03:25:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa12, chip 3
2018-11-20 03:25:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:25:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:25:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:25:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:25:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:25:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:25:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:25:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:25:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:25:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:25:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:25:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:25:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:25:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:25:55 [DEBUG   ] Finished updating board flags in 1.813 seconds
2018-11-20 03:25:56 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 03:26:16 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:26:36 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:26:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:26:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:26:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:26:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:26:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:26:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-20 03:26:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:26:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:26:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:26:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:26:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:26:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:26:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:26:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:26:55 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:26:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 03:26:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:26:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:26:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:26:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:26:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:26:55 [DEBUG   ] Finished updating board flags in 1.712 seconds
2018-11-20 03:26:56 [DEBUG   ] Finished updating ant stats in 0.797 seconds
2018-11-20 03:27:16 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 03:27:36 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 03:27:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:27:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366157492000000
2018-11-20 03:27:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:27:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:27:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:27:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:27:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366157492000000
2018-11-20 03:27:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:27:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:27:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:27:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:27:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:27:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:27:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:27:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:27:55 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:27:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:27:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:27:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:27:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:27:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:27:55 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:27:55 [DEBUG   ] Finished updating board flags in 1.584 seconds
2018-11-20 03:27:56 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 03:28:16 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 03:28:36 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 03:28:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:28:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366169252000000
2018-11-20 03:28:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:28:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:28:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:28:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:28:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366169252000000
2018-11-20 03:28:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 03:28:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:28:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:28:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:28:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:28:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:28:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:28:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:28:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-20 03:28:55 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:28:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:28:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:28:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:28:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:28:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:28:55 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:28:55 [DEBUG   ] Finished updating board flags in 1.441 seconds
2018-11-20 03:28:56 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 03:29:16 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 03:29:36 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 03:29:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:29:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366181012000000
2018-11-20 03:29:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:29:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:29:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:29:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:29:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366181012000000
2018-11-20 03:29:54 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:29:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:29:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:29:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:29:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:29:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:29:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 03:29:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:29:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:29:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:29:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:29:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:29:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:29:55 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:29:55 [DEBUG   ] Finished updating board flags in 1.467 seconds
2018-11-20 03:29:56 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 03:30:06 [DEBUG   ] Got command RPT from MCS: ref #565194
2018-11-20 03:30:06 [DEBUG   ] summary = ERROR
2018-11-20 03:30:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565194  13 58442 12606665 A  ERRORERROR'
2018-11-20 03:30:06 [DEBUG   ] Got command RPT from MCS: ref #565195
2018-11-20 03:30:06 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 03:30:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565195  71 58442 12606869 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 03:30:07 [DEBUG   ] Got command RPT from MCS: ref #565196
2018-11-20 03:30:07 [DEBUG   ] summary = ERROR
2018-11-20 03:30:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565196  13 58442 12607061 A  ERRORERROR'
2018-11-20 03:30:07 [DEBUG   ] Got command RPT from MCS: ref #565198
2018-11-20 03:30:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 03:30:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565198  71 58442 12607265 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 03:30:07 [DEBUG   ] Got command RPT from MCS: ref #565199
2018-11-20 03:30:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:30:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565199  12 58442 12607470 A  ERRORB¤ff'
2018-11-20 03:30:16 [DEBUG   ] Finished updating ant stats in 0.594 seconds
2018-11-20 03:30:36 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 03:30:46 [DEBUG   ] Got command RPT from MCS: ref #565204
2018-11-20 03:30:46 [DEBUG   ] summary = ERROR
2018-11-20 03:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565204  13 58442 12646593 A  ERRORERROR'
2018-11-20 03:30:46 [DEBUG   ] Got command RPT from MCS: ref #565205
2018-11-20 03:30:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 03:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565205  71 58442 12646594 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 03:30:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:30:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366192772000000
2018-11-20 03:30:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:30:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:30:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:30:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:30:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366192772000000
2018-11-20 03:30:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 03:30:54 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 03:30:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:30:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:30:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366192968000000
2018-11-20 03:30:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:30:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:30:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:30:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:30:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366192968000000
2018-11-20 03:30:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 03:30:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:30:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:30:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:30:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:30:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:30:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:30:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:30:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:30:55 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:30:55 [DEBUG   ] Finished updating board flags in 1.464 seconds
2018-11-20 03:30:57 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 03:31:17 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 03:31:37 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 03:31:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:31:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366204532000000
2018-11-20 03:31:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:31:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:31:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:31:54 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:31:54 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366204532000000
2018-11-20 03:31:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:31:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:31:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:31:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366204728000000
2018-11-20 03:31:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:31:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:31:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:31:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:31:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366204728000000
2018-11-20 03:31:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 03:31:55 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:31:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:31:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:31:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:31:55 [WARNING ] (13, 1, 'bot')
2018-11-20 03:31:55 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:31:55 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:31:55 [DEBUG   ] Finished updating board flags in 1.459 seconds
2018-11-20 03:31:57 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 03:32:17 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 03:32:37 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:32:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:32:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:32:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:32:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:32:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:32:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:32:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:32:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:32:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:32:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:32:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:32:56 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:32:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:32:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:32:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:32:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:32:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:32:56 [DEBUG   ] Finished updating board flags in 2.279 seconds
2018-11-20 03:32:57 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:33:17 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:33:37 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:33:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:33:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:33:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:33:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:33:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:33:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:33:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:33:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:33:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:33:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:33:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:33:56 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:33:56 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:33:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:33:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:33:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:33:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:33:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:33:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:33:56 [DEBUG   ] Finished updating board flags in 2.172 seconds
2018-11-20 03:33:57 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:34:17 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:34:37 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:34:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:34:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:34:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:34:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:34:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:34:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 03:34:55 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 03:34:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:34:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:34:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:34:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:34:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:34:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:34:56 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:34:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:34:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:34:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:34:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:34:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:34:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:34:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:34:56 [DEBUG   ] Finished updating board flags in 1.983 seconds
2018-11-20 03:34:57 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 03:35:06 [DEBUG   ] Got command RPT from MCS: ref #565229
2018-11-20 03:35:06 [DEBUG   ] summary = ERROR
2018-11-20 03:35:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565229  13 58442 12906745 A  ERRORERROR'
2018-11-20 03:35:06 [DEBUG   ] Got command RPT from MCS: ref #565230
2018-11-20 03:35:06 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:35:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565230  98 58442 12906949 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:35:07 [DEBUG   ] Got command RPT from MCS: ref #565231
2018-11-20 03:35:07 [DEBUG   ] summary = ERROR
2018-11-20 03:35:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565231  13 58442 12907141 A  ERRORERROR'
2018-11-20 03:35:07 [DEBUG   ] Got command RPT from MCS: ref #565232
2018-11-20 03:35:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:35:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565232  98 58442 12907355 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:35:07 [DEBUG   ] Got command RPT from MCS: ref #565233
2018-11-20 03:35:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:35:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565233  12 58442 12907545 A  ERRORB£ff'
2018-11-20 03:35:17 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:35:37 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:35:46 [DEBUG   ] Got command RPT from MCS: ref #565234
2018-11-20 03:35:46 [DEBUG   ] summary = ERROR
2018-11-20 03:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565234  13 58442 12946593 A  ERRORERROR'
2018-11-20 03:35:46 [DEBUG   ] Got command RPT from MCS: ref #565235
2018-11-20 03:35:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565235  98 58442 12946594 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:35:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:35:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:35:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:35:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:35:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:35:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:35:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:35:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:35:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:35:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:35:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:35:56 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:35:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:35:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:35:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:35:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:35:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:35:56 [DEBUG   ] Finished updating board flags in 1.831 seconds
2018-11-20 03:35:57 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:36:17 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 03:36:37 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 03:36:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:36:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:36:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:36:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:36:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:36:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:36:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-20 03:36:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:36:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:36:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:36:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:36:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:36:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:36:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 03:36:56 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:36:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:36:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:36:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:36:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:36:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:36:56 [DEBUG   ] Finished updating board flags in 1.794 seconds
2018-11-20 03:36:57 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 03:37:17 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:37:37 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 03:37:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:37:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366275288000000
2018-11-20 03:37:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:37:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:37:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:37:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:37:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366275288000000
2018-11-20 03:37:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-20 03:37:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 03:37:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:37:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 03:37:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:37:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:37:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:37:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:37:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:37:56 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:37:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:37:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:37:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:37:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:37:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:37:56 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:37:56 [DEBUG   ] Finished updating board flags in 1.581 seconds
2018-11-20 03:37:57 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 03:38:17 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 03:38:37 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 03:38:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:38:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366287048000000
2018-11-20 03:38:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:38:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:38:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:38:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:38:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366287048000000
2018-11-20 03:38:55 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:38:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 03:38:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:38:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366287244000000
2018-11-20 03:38:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:38:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:38:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:38:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:38:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366287244000000
2018-11-20 03:38:56 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:38:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:38:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:38:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:38:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:38:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:38:56 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:38:56 [DEBUG   ] Finished updating board flags in 1.439 seconds
2018-11-20 03:38:57 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 03:39:17 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 03:39:38 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 03:39:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:39:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366298808000000
2018-11-20 03:39:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:39:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:39:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:39:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:39:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366298808000000
2018-11-20 03:39:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:39:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:39:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366299004000000
2018-11-20 03:39:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:39:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:39:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:39:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:39:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366299004000000
2018-11-20 03:39:56 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:39:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:39:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:39:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:39:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:39:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:39:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:39:56 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:39:56 [DEBUG   ] Finished updating board flags in 1.451 seconds
2018-11-20 03:39:58 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 03:40:06 [DEBUG   ] Got command RPT from MCS: ref #565257
2018-11-20 03:40:06 [DEBUG   ] summary = ERROR
2018-11-20 03:40:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565257  13 58442 13206829 A  ERRORERROR'
2018-11-20 03:40:07 [DEBUG   ] Got command RPT from MCS: ref #565258
2018-11-20 03:40:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:40:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565258  79 58442 13207021 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:40:07 [DEBUG   ] Got command RPT from MCS: ref #565259
2018-11-20 03:40:07 [DEBUG   ] summary = ERROR
2018-11-20 03:40:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565259  13 58442 13207225 A  ERRORERROR'
2018-11-20 03:40:07 [DEBUG   ] Got command RPT from MCS: ref #565261
2018-11-20 03:40:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:40:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565261  79 58442 13207429 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:40:07 [DEBUG   ] Got command RPT from MCS: ref #565262
2018-11-20 03:40:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:40:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565262  12 58442 13207633 A  ERRORB¤ff'
2018-11-20 03:40:18 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 03:40:38 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 03:40:46 [DEBUG   ] Got command RPT from MCS: ref #565271
2018-11-20 03:40:46 [DEBUG   ] summary = ERROR
2018-11-20 03:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565271  13 58442 13246589 A  ERRORERROR'
2018-11-20 03:40:46 [DEBUG   ] Got command RPT from MCS: ref #565272
2018-11-20 03:40:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565272  79 58442 13246590 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:40:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:40:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366310568000000
2018-11-20 03:40:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:40:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:40:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:40:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:40:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366310568000000
2018-11-20 03:40:55 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:40:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:40:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:40:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366310764000000
2018-11-20 03:40:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:40:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:40:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:40:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:40:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366310764000000
2018-11-20 03:40:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 03:40:56 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:40:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:40:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:40:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:40:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:40:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:40:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:40:56 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:40:56 [DEBUG   ] Finished updating board flags in 1.450 seconds
2018-11-20 03:40:58 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 03:41:18 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 03:41:38 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 03:41:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:41:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366322328000000
2018-11-20 03:41:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:41:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:41:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:41:55 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:41:55 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366322328000000
2018-11-20 03:41:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:41:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:41:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366322524000000
2018-11-20 03:41:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:41:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:41:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:41:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:41:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366322524000000
2018-11-20 03:41:56 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:41:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:41:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:41:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:41:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:41:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:41:56 [WARNING ] (13, 1, 'bot')
2018-11-20 03:41:56 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:41:56 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:41:56 [DEBUG   ] Finished updating board flags in 1.503 seconds
2018-11-20 03:41:58 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 03:42:18 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:42:38 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 03:42:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:42:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:42:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:42:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:42:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:42:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:42:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:42:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:42:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:42:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:42:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:42:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:42:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:42:58 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:42:58 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 03:42:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:42:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:42:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:42:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:42:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:42:58 [DEBUG   ] Finished updating board flags in 3.022 seconds
2018-11-20 03:43:18 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:43:38 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:43:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:43:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:43:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:43:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:43:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:43:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 03:43:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:43:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:43:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:43:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:43:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:43:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:43:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 03:43:57 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:43:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:43:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:43:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:43:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:43:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:43:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:43:57 [DEBUG   ] Finished updating board flags in 2.220 seconds
2018-11-20 03:43:58 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:44:18 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:44:38 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:44:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:44:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:44:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:44:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:44:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:44:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 03:44:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 03:44:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:44:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:44:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:44:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:44:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:44:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:44:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:44:57 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:44:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:44:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:44:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:44:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:44:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:44:57 [DEBUG   ] Finished updating board flags in 2.126 seconds
2018-11-20 03:44:58 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:45:06 [DEBUG   ] Got command RPT from MCS: ref #565292
2018-11-20 03:45:06 [DEBUG   ] summary = ERROR
2018-11-20 03:45:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565292  13 58442 13506909 A  ERRORERROR'
2018-11-20 03:45:07 [DEBUG   ] Got command RPT from MCS: ref #565293
2018-11-20 03:45:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:45:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565293  98 58442 13507113 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:45:07 [DEBUG   ] Got command RPT from MCS: ref #565294
2018-11-20 03:45:07 [DEBUG   ] summary = ERROR
2018-11-20 03:45:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565294  13 58442 13507305 A  ERRORERROR'
2018-11-20 03:45:07 [DEBUG   ] Got command RPT from MCS: ref #565295
2018-11-20 03:45:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:45:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565295  98 58442 13507509 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:45:07 [DEBUG   ] Got command RPT from MCS: ref #565296
2018-11-20 03:45:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:45:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565296  12 58442 13507713 A  ERRORB£ff'
2018-11-20 03:45:18 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 03:45:38 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:45:46 [DEBUG   ] Got command RPT from MCS: ref #565297
2018-11-20 03:45:46 [DEBUG   ] summary = ERROR
2018-11-20 03:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565297  13 58442 13546589 A  ERRORERROR'
2018-11-20 03:45:46 [DEBUG   ] Got command RPT from MCS: ref #565298
2018-11-20 03:45:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 03:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565298  98 58442 13546590 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 03:45:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:45:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:45:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:45:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:45:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:45:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:45:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 03:45:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:45:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:45:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:45:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:45:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:45:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:45:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:45:57 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:45:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:45:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:45:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:45:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:45:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:45:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:45:57 [DEBUG   ] Finished updating board flags in 2.029 seconds
2018-11-20 03:45:58 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:46:18 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:46:38 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 03:46:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:46:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:46:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:46:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:46:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:46:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:46:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:46:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:46:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:46:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:46:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:46:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:46:57 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:46:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:46:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:46:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:46:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:46:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:46:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:46:57 [DEBUG   ] Finished updating board flags in 1.956 seconds
2018-11-20 03:46:58 [DEBUG   ] Finished updating ant stats in 0.793 seconds
2018-11-20 03:47:18 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 03:47:38 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 03:47:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:47:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:47:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:47:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:47:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:47:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 03:47:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:47:56 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:47:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:47:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:47:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:47:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:47:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:47:57 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:47:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:47:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:47:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:47:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:47:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:47:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:47:57 [DEBUG   ] Finished updating board flags in 1.815 seconds
2018-11-20 03:47:58 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 03:48:18 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 03:48:38 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 03:48:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:48:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:48:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:48:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:48:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:48:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:48:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:48:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:48:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:48:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:48:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:48:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:48:57 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:48:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:48:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:48:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:48:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:48:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:48:57 [DEBUG   ] Finished updating board flags in 1.771 seconds
2018-11-20 03:48:58 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 03:49:18 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 03:49:38 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 03:49:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:49:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366416604000000
2018-11-20 03:49:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:49:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:49:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:49:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:49:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366416604000000
2018-11-20 03:49:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa10, chip 2
2018-11-20 03:49:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:49:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:49:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:49:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:49:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:49:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:49:57 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:49:57 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:49:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:49:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:49:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:49:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:49:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:49:57 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:49:57 [DEBUG   ] Finished updating board flags in 1.687 seconds
2018-11-20 03:49:58 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 03:50:06 [DEBUG   ] Got command RPT from MCS: ref #565324
2018-11-20 03:50:06 [DEBUG   ] summary = ERROR
2018-11-20 03:50:06 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565324  13 58442 13806985 A  ERRORERROR'
2018-11-20 03:50:07 [DEBUG   ] Got command RPT from MCS: ref #565325
2018-11-20 03:50:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 03:50:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565325  71 58442 13807189 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 03:50:07 [DEBUG   ] Got command RPT from MCS: ref #565326
2018-11-20 03:50:07 [DEBUG   ] summary = ERROR
2018-11-20 03:50:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565326  13 58442 13807385 A  ERRORERROR'
2018-11-20 03:50:07 [DEBUG   ] Got command RPT from MCS: ref #565328
2018-11-20 03:50:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 03:50:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565328  71 58442 13807589 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 03:50:07 [DEBUG   ] Got command RPT from MCS: ref #565329
2018-11-20 03:50:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:50:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565329  12 58442 13807793 A  ERRORB¤ff'
2018-11-20 03:50:18 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 03:50:38 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 03:50:46 [DEBUG   ] Got command RPT from MCS: ref #565334
2018-11-20 03:50:46 [DEBUG   ] summary = ERROR
2018-11-20 03:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565334  13 58442 13846593 A  ERRORERROR'
2018-11-20 03:50:46 [DEBUG   ] Got command RPT from MCS: ref #565335
2018-11-20 03:50:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 03:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565335  71 58442 13846594 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 03:50:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:50:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366428364000000
2018-11-20 03:50:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:50:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:50:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:50:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:50:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366428364000000
2018-11-20 03:50:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 03:50:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:50:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:50:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:50:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:50:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:50:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:50:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:50:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:50:57 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:50:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:50:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:50:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:50:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:50:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:50:57 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:50:57 [DEBUG   ] Finished updating board flags in 1.508 seconds
2018-11-20 03:50:59 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 03:51:19 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 03:51:39 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 03:51:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:51:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366440124000000
2018-11-20 03:51:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:51:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:51:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:51:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:51:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366440124000000
2018-11-20 03:51:56 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:51:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:51:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:51:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:51:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:51:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:51:57 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:51:57 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:51:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:51:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:51:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:51:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:51:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:51:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:51:57 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 03:51:57 [DEBUG   ] Finished updating board flags in 1.455 seconds
2018-11-20 03:51:59 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 03:52:19 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 03:52:39 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 03:52:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:52:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366451884000000
2018-11-20 03:52:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:52:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:52:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:52:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:52:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366451884000000
2018-11-20 03:52:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 03:52:56 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 03:52:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:52:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 03:52:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:52:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366452080000000
2018-11-20 03:52:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:52:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:52:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:52:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:52:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366452080000000
2018-11-20 03:52:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 03:52:57 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:52:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:52:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:52:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:52:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:52:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:52:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:52:57 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:52:57 [DEBUG   ] Finished updating board flags in 1.481 seconds
2018-11-20 03:52:59 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 03:53:19 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:53:39 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 03:53:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:53:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366463644000000
2018-11-20 03:53:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:53:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:53:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:53:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:53:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366463644000000
2018-11-20 03:53:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:53:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:53:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366463840000000
2018-11-20 03:53:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:53:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:53:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:53:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:53:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366463840000000
2018-11-20 03:53:57 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:53:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:53:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:53:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:53:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:53:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:53:57 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:53:57 [DEBUG   ] Finished updating board flags in 1.467 seconds
2018-11-20 03:53:59 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:54:19 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:54:39 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:54:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:54:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366475404000000
2018-11-20 03:54:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:54:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:54:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:54:56 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:54:56 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366475404000000
2018-11-20 03:54:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:54:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:54:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366475600000000
2018-11-20 03:54:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:54:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:54:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:54:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:54:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366475600000000
2018-11-20 03:54:57 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:54:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 03:54:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:54:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:54:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:54:57 [WARNING ] (13, 1, 'bot')
2018-11-20 03:54:57 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:54:57 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 03:54:57 [DEBUG   ] Finished updating board flags in 1.476 seconds
2018-11-20 03:54:59 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:55:07 [DEBUG   ] Got command RPT from MCS: ref #565355
2018-11-20 03:55:07 [DEBUG   ] summary = ERROR
2018-11-20 03:55:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565355  13 58442 14107069 A  ERRORERROR'
2018-11-20 03:55:07 [DEBUG   ] Got command RPT from MCS: ref #565356
2018-11-20 03:55:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:55:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565356  79 58442 14107274 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:55:07 [DEBUG   ] Got command RPT from MCS: ref #565357
2018-11-20 03:55:07 [DEBUG   ] summary = ERROR
2018-11-20 03:55:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565357  13 58442 14107465 A  ERRORERROR'
2018-11-20 03:55:07 [DEBUG   ] Got command RPT from MCS: ref #565358
2018-11-20 03:55:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:55:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565358  79 58442 14107669 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:55:07 [DEBUG   ] Got command RPT from MCS: ref #565359
2018-11-20 03:55:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 03:55:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565359  12 58442 14107873 A  ERRORB¤ff'
2018-11-20 03:55:19 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:55:39 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:55:46 [DEBUG   ] Got command RPT from MCS: ref #565364
2018-11-20 03:55:46 [DEBUG   ] summary = ERROR
2018-11-20 03:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565364  13 58442 14146589 A  ERRORERROR'
2018-11-20 03:55:46 [DEBUG   ] Got command RPT from MCS: ref #565365
2018-11-20 03:55:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 03:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565365  79 58442 14146590 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 03:55:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:55:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:55:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:55:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:55:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:55:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 03:55:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:55:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:55:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:55:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:55:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:55:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:55:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:55:58 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:55:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:55:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:55:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:55:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:55:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:55:58 [DEBUG   ] Finished updating board flags in 2.230 seconds
2018-11-20 03:55:59 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 03:56:19 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 03:56:39 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 03:56:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:56:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:56:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:56:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:56:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:56:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 03:56:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:56:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:56:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:56:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:56:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:56:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:56:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 03:56:58 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 03:56:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:56:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:56:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:56:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:56:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:56:58 [DEBUG   ] Finished updating board flags in 2.208 seconds
2018-11-20 03:56:59 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 03:57:19 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:57:39 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:57:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:57:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:57:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:57:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:57:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:57:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:57:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:57:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:57:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:57:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:57:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:57:58 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:57:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 03:57:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:57:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:57:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:57:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:57:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:57:58 [DEBUG   ] Finished updating board flags in 2.163 seconds
2018-11-20 03:57:59 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 03:58:19 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:58:39 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:58:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:58:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:58:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:58:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:58:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:58:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:58:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 03:58:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:58:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:58:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:58:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:58:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:58:58 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:58:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:58:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:58:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:58:58 [WARNING ] (13, 1, 'bot')
2018-11-20 03:58:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:58:58 [DEBUG   ] Finished updating board flags in 1.985 seconds
2018-11-20 03:58:59 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 03:59:19 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 03:59:39 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 03:59:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 03:59:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 03:59:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 03:59:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 03:59:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 03:59:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 03:59:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 03:59:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 03:59:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 03:59:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 03:59:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 03:59:58 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 03:59:58 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 03:59:59 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 03:59:59 [WARNING ] (13, 1, 'bot')
2018-11-20 03:59:59 [WARNING ] (13, 1, 'bot')
2018-11-20 03:59:59 [WARNING ] (13, 1, 'bot')
2018-11-20 03:59:59 [WARNING ] (13, 1, 'bot')
2018-11-20 03:59:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 03:59:59 [DEBUG   ] Finished updating board flags in 2.513 seconds
2018-11-20 04:00:07 [DEBUG   ] Got command RPT from MCS: ref #565387
2018-11-20 04:00:07 [DEBUG   ] summary = ERROR
2018-11-20 04:00:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565387  13 58442 14407149 A  ERRORERROR'
2018-11-20 04:00:07 [DEBUG   ] Got command RPT from MCS: ref #565388
2018-11-20 04:00:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:00:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565388  98 58442 14407354 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:00:07 [DEBUG   ] Got command RPT from MCS: ref #565389
2018-11-20 04:00:07 [DEBUG   ] summary = ERROR
2018-11-20 04:00:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565389  13 58442 14407545 A  ERRORERROR'
2018-11-20 04:00:07 [DEBUG   ] Got command RPT from MCS: ref #565391
2018-11-20 04:00:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:00:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565391  98 58442 14407753 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:00:07 [DEBUG   ] Got command RPT from MCS: ref #565392
2018-11-20 04:00:07 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:00:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565392  12 58442 14407945 A  ERRORB£ff'
2018-11-20 04:00:19 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 04:00:39 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 04:00:46 [DEBUG   ] Got command RPT from MCS: ref #565397
2018-11-20 04:00:46 [DEBUG   ] summary = ERROR
2018-11-20 04:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565397  13 58442 14446593 A  ERRORERROR'
2018-11-20 04:00:46 [DEBUG   ] Got command RPT from MCS: ref #565398
2018-11-20 04:00:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565398  98 58442 14446594 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:00:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:00:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:00:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:00:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:00:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:00:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:00:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-20 04:00:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:00:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:00:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:00:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:00:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:00:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:00:58 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:00:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:00:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:00:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:00:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:00:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:00:58 [DEBUG   ] Finished updating board flags in 1.641 seconds
2018-11-20 04:00:59 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 04:01:19 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 04:01:39 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 04:01:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:01:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366557920000000
2018-11-20 04:01:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:01:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:01:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:01:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:01:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366557920000000
2018-11-20 04:01:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:01:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:01:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:01:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:01:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:01:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:01:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-20 04:01:58 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:01:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:01:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:01:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:01:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:01:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:01:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:01:58 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:01:58 [DEBUG   ] Finished updating board flags in 1.617 seconds
2018-11-20 04:01:59 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 04:02:19 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 04:02:39 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 04:02:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:02:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366569680000000
2018-11-20 04:02:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:02:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:02:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:02:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:02:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366569680000000
2018-11-20 04:02:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 04:02:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 04:02:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:02:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 04:02:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:02:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:02:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:02:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:02:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:02:58 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:02:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:02:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:02:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:02:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:02:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:02:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:02:58 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:02:58 [DEBUG   ] Finished updating board flags in 1.589 seconds
2018-11-20 04:03:00 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 04:03:20 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 04:03:40 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 04:03:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:03:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366581440000000
2018-11-20 04:03:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:03:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:03:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:03:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:03:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366581440000000
2018-11-20 04:03:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 04:03:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:03:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:03:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:03:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:03:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:03:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:03:58 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:03:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:03:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:03:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:03:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:03:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:03:58 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:03:58 [DEBUG   ] Finished updating board flags in 1.415 seconds
2018-11-20 04:04:00 [DEBUG   ] Finished updating ant stats in 1.194 seconds
2018-11-20 04:04:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:04:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:04:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:04:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366593200000000
2018-11-20 04:04:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:04:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:04:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:04:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:04:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366593200000000
2018-11-20 04:04:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 04:04:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 04:04:57 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:04:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:04:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:04:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:04:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:04:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:04:58 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:04:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:04:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:04:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:04:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:04:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:04:58 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:04:58 [DEBUG   ] Finished updating board flags in 1.376 seconds
2018-11-20 04:05:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:05:07 [DEBUG   ] Got command RPT from MCS: ref #565422
2018-11-20 04:05:07 [DEBUG   ] summary = ERROR
2018-11-20 04:05:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565422  13 58442 14707229 A  ERRORERROR'
2018-11-20 04:05:07 [DEBUG   ] Got command RPT from MCS: ref #565423
2018-11-20 04:05:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:05:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565423  71 58442 14707433 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:05:07 [DEBUG   ] Got command RPT from MCS: ref #565424
2018-11-20 04:05:07 [DEBUG   ] summary = ERROR
2018-11-20 04:05:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565424  13 58442 14707625 A  ERRORERROR'
2018-11-20 04:05:07 [DEBUG   ] Got command RPT from MCS: ref #565425
2018-11-20 04:05:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:05:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565425  71 58442 14707829 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:05:08 [DEBUG   ] Got command RPT from MCS: ref #565426
2018-11-20 04:05:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:05:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565426  12 58442 14708033 A  ERRORB¤ff'
2018-11-20 04:05:20 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:05:40 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:05:46 [DEBUG   ] Got command RPT from MCS: ref #565427
2018-11-20 04:05:46 [DEBUG   ] summary = ERROR
2018-11-20 04:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565427  13 58442 14746593 A  ERRORERROR'
2018-11-20 04:05:46 [DEBUG   ] Got command RPT from MCS: ref #565428
2018-11-20 04:05:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565428  71 58442 14746594 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:05:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:05:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366604960000000
2018-11-20 04:05:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:05:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:05:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:05:57 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:05:57 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366604960000000
2018-11-20 04:05:57 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 04:05:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 04:05:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:05:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:05:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366605156000000
2018-11-20 04:05:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:05:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:05:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:05:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:05:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366605156000000
2018-11-20 04:05:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 04:05:58 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:05:58 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:05:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:05:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:05:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:05:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:05:58 [WARNING ] (13, 1, 'bot')
2018-11-20 04:05:58 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:05:58 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 04:05:58 [DEBUG   ] Finished updating board flags in 1.499 seconds
2018-11-20 04:06:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:06:20 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:06:40 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:06:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:06:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:06:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:06:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:06:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:06:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:06:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 04:06:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:06:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:06:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:06:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:06:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:06:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:06:59 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:06:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:06:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:06:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:06:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:06:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:06:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:06:59 [DEBUG   ] Finished updating board flags in 2.219 seconds
2018-11-20 04:07:00 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 04:07:20 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 04:07:40 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:07:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:07:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:07:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:07:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:07:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:07:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:07:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:07:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:07:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:07:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:07:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:07:59 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:08:00 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:08:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:08:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:08:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:08:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:08:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:08:00 [DEBUG   ] Finished updating board flags in 2.859 seconds
2018-11-20 04:08:20 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:08:40 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 04:08:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:08:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:08:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:08:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:08:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:08:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 04:08:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:08:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:08:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:08:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:08:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:08:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:08:59 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:08:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:08:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:08:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:08:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:08:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:08:59 [DEBUG   ] Finished updating board flags in 2.121 seconds
2018-11-20 04:09:00 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 04:09:20 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:09:40 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 04:09:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:09:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:09:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:09:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:09:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:09:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:09:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:09:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:09:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:09:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:09:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:09:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:09:59 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:09:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:09:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:09:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:09:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:09:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:09:59 [DEBUG   ] Finished updating board flags in 1.934 seconds
2018-11-20 04:10:00 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 04:10:07 [DEBUG   ] Got command RPT from MCS: ref #565450
2018-11-20 04:10:07 [DEBUG   ] summary = ERROR
2018-11-20 04:10:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565450  13 58442 15007305 A  ERRORERROR'
2018-11-20 04:10:07 [DEBUG   ] Got command RPT from MCS: ref #565451
2018-11-20 04:10:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:10:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565451  98 58442 15007509 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:10:07 [DEBUG   ] Got command RPT from MCS: ref #565452
2018-11-20 04:10:07 [DEBUG   ] summary = ERROR
2018-11-20 04:10:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565452  13 58442 15007713 A  ERRORERROR'
2018-11-20 04:10:07 [DEBUG   ] Got command RPT from MCS: ref #565454
2018-11-20 04:10:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:10:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565454  98 58442 15007918 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:10:08 [DEBUG   ] Got command RPT from MCS: ref #565456
2018-11-20 04:10:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:10:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565456  12 58442 15008109 A  ERRORB£ff'
2018-11-20 04:10:20 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 04:10:40 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 04:10:46 [DEBUG   ] Got command RPT from MCS: ref #565464
2018-11-20 04:10:46 [DEBUG   ] summary = ERROR
2018-11-20 04:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565464  13 58442 15046597 A  ERRORERROR'
2018-11-20 04:10:46 [DEBUG   ] Got command RPT from MCS: ref #565465
2018-11-20 04:10:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565465  98 58442 15046598 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:10:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:10:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:10:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:10:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:10:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:10:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 04:10:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:10:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:10:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:10:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:10:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:10:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:10:59 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:10:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:10:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:10:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:10:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:10:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:10:59 [DEBUG   ] Finished updating board flags in 1.825 seconds
2018-11-20 04:11:01 [DEBUG   ] Finished updating ant stats in 1.402 seconds
2018-11-20 04:11:20 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 04:11:40 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 04:11:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:11:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:11:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:11:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:11:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:11:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 04:11:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:11:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 04:11:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:11:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:11:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:11:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:11:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:11:59 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:11:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:11:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:11:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:11:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:11:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:11:59 [DEBUG   ] Finished updating board flags in 1.772 seconds
2018-11-20 04:12:01 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 04:12:21 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 04:12:41 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 04:12:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:12:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366687476000000
2018-11-20 04:12:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:12:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:12:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:12:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:12:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366687476000000
2018-11-20 04:12:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:12:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 04:12:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 04:12:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:12:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:12:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:12:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:12:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:12:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:12:59 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:12:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:12:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:12:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:12:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:12:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:12:59 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:12:59 [DEBUG   ] Finished updating board flags in 1.611 seconds
2018-11-20 04:13:01 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 04:13:21 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 04:13:41 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 04:13:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:13:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366699236000000
2018-11-20 04:13:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:13:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:13:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:13:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:13:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366699236000000
2018-11-20 04:13:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:13:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:13:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 04:13:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:13:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:13:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:13:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:13:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:13:59 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:13:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:13:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:13:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:13:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:13:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:13:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:13:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:13:59 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:13:59 [DEBUG   ] Finished updating board flags in 1.592 seconds
2018-11-20 04:14:01 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:14:21 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:14:41 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:14:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:14:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366710996000000
2018-11-20 04:14:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:14:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:14:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:14:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:14:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366710996000000
2018-11-20 04:14:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 04:14:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:14:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:14:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:14:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:14:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:14:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:14:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 04:14:59 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:14:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 04:14:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:14:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:14:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:14:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:14:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:14:59 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:14:59 [DEBUG   ] Finished updating board flags in 1.414 seconds
2018-11-20 04:15:01 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:15:07 [DEBUG   ] Got command RPT from MCS: ref #565485
2018-11-20 04:15:07 [DEBUG   ] summary = ERROR
2018-11-20 04:15:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565485  13 58442 15307393 A  ERRORERROR'
2018-11-20 04:15:07 [DEBUG   ] Got command RPT from MCS: ref #565486
2018-11-20 04:15:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:15:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565486  71 58442 15307585 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:15:07 [DEBUG   ] Got command RPT from MCS: ref #565487
2018-11-20 04:15:07 [DEBUG   ] summary = ERROR
2018-11-20 04:15:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565487  13 58442 15307789 A  ERRORERROR'
2018-11-20 04:15:07 [DEBUG   ] Got command RPT from MCS: ref #565488
2018-11-20 04:15:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:15:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565488  71 58442 15307993 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:15:08 [DEBUG   ] Got command RPT from MCS: ref #565489
2018-11-20 04:15:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:15:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565489  12 58442 15308197 A  ERRORB£ff'
2018-11-20 04:15:21 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:15:41 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:15:46 [DEBUG   ] Got command RPT from MCS: ref #565490
2018-11-20 04:15:46 [DEBUG   ] summary = ERROR
2018-11-20 04:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565490  13 58442 15346597 A  ERRORERROR'
2018-11-20 04:15:46 [DEBUG   ] Got command RPT from MCS: ref #565491
2018-11-20 04:15:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565491  71 58442 15346598 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:15:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:15:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366722756000000
2018-11-20 04:15:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:15:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:15:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:15:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:15:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366722756000000
2018-11-20 04:15:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:15:58 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 04:15:58 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:15:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:15:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366722952000000
2018-11-20 04:15:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:15:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:15:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:15:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:15:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366722952000000
2018-11-20 04:15:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 04:15:59 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:15:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:15:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:15:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:15:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:15:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:15:59 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 04:15:59 [DEBUG   ] Finished updating board flags in 1.524 seconds
2018-11-20 04:16:01 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:16:21 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:16:41 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:16:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:16:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366734516000000
2018-11-20 04:16:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:16:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:16:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:16:58 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:16:58 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366734516000000
2018-11-20 04:16:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:16:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:16:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366734712000000
2018-11-20 04:16:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:16:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:16:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:16:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:16:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366734712000000
2018-11-20 04:16:59 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:16:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:16:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:16:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:16:59 [WARNING ] (13, 1, 'bot')
2018-11-20 04:16:59 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:16:59 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 04:16:59 [DEBUG   ] Finished updating board flags in 1.538 seconds
2018-11-20 04:17:01 [DEBUG   ] Finished updating ant stats in 1.073 seconds
2018-11-20 04:17:21 [DEBUG   ] Finished updating ant stats in 0.932 seconds
2018-11-20 04:17:41 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 04:17:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:17:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:17:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:17:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:17:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:17:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 04:17:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:17:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 04:18:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:18:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:18:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:18:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:18:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:18:00 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:18:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:18:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:18:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:18:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:18:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:18:00 [DEBUG   ] Finished updating board flags in 2.223 seconds
2018-11-20 04:18:01 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:18:21 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:18:41 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:18:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:18:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:18:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:18:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:18:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:18:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 04:18:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 04:18:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:19:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:19:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:19:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:19:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:19:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:19:00 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:19:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:19:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:19:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:19:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:19:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:19:00 [DEBUG   ] Finished updating board flags in 2.160 seconds
2018-11-20 04:19:01 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 04:19:21 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 04:19:41 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 04:19:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:19:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:19:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:19:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:19:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:19:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:20:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:20:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:20:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:20:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:20:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:20:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 04:20:00 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:20:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:20:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:20:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:20:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:20:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:20:00 [DEBUG   ] Finished updating board flags in 2.004 seconds
2018-11-20 04:20:01 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 04:20:07 [DEBUG   ] Got command RPT from MCS: ref #565517
2018-11-20 04:20:07 [DEBUG   ] summary = ERROR
2018-11-20 04:20:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565517  13 58442 15607469 A  ERRORERROR'
2018-11-20 04:20:07 [DEBUG   ] Got command RPT from MCS: ref #565518
2018-11-20 04:20:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:20:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565518  98 58442 15607673 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:20:07 [DEBUG   ] Got command RPT from MCS: ref #565519
2018-11-20 04:20:07 [DEBUG   ] summary = ERROR
2018-11-20 04:20:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565519  13 58442 15607869 A  ERRORERROR'
2018-11-20 04:20:08 [DEBUG   ] Got command RPT from MCS: ref #565521
2018-11-20 04:20:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:20:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565521  98 58442 15608073 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:20:08 [DEBUG   ] Got command RPT from MCS: ref #565522
2018-11-20 04:20:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:20:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565522  12 58442 15608277 A  ERRORB£ff'
2018-11-20 04:20:21 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 04:20:41 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-20 04:20:46 [DEBUG   ] Got command RPT from MCS: ref #565527
2018-11-20 04:20:46 [DEBUG   ] summary = ERROR
2018-11-20 04:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565527  13 58442 15646593 A  ERRORERROR'
2018-11-20 04:20:46 [DEBUG   ] Got command RPT from MCS: ref #565528
2018-11-20 04:20:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565528  98 58442 15646594 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:20:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:20:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:20:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:20:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:20:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:20:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:21:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:21:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:21:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:21:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:21:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:21:00 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:21:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:21:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:21:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:21:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:21:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:21:00 [DEBUG   ] Finished updating board flags in 1.932 seconds
2018-11-20 04:21:02 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 04:21:21 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-20 04:21:42 [DEBUG   ] Finished updating ant stats in 1.394 seconds
2018-11-20 04:21:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:21:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:21:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:21:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:21:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:21:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:21:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 04:21:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 04:21:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:21:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 04:22:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:22:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:22:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:22:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:22:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:22:00 [WARNING ] MonitorFlags: Temperature 82.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:22:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:22:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:22:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:22:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:22:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:22:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:22:00 [DEBUG   ] Finished updating board flags in 1.813 seconds
2018-11-20 04:22:02 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 04:22:22 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 04:22:42 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-20 04:22:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:22:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:22:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:22:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:22:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:22:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:23:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:23:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:23:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:23:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:23:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:23:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 04:23:00 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:23:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:23:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:23:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:23:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:23:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:23:00 [DEBUG   ] Finished updating board flags in 1.784 seconds
2018-11-20 04:23:02 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:23:22 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:23:42 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:23:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:23:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366817032000000
2018-11-20 04:23:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:23:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:23:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:23:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:23:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366817032000000
2018-11-20 04:23:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:23:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 04:23:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 04:23:59 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 04:23:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:24:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:24:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:24:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:24:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:24:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:24:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-20 04:24:00 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:24:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:24:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:24:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:24:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:24:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:24:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:24:00 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:24:00 [DEBUG   ] Finished updating board flags in 1.622 seconds
2018-11-20 04:24:02 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 04:24:22 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:24:42 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:24:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:24:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366828792000000
2018-11-20 04:24:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:24:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:24:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:24:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:24:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366828792000000
2018-11-20 04:24:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 04:24:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 04:24:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:25:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:25:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:25:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:25:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:25:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:25:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 04:25:00 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:25:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:25:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:25:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:25:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:25:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:25:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:25:00 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:25:00 [DEBUG   ] Finished updating board flags in 1.592 seconds
2018-11-20 04:25:02 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 04:25:07 [DEBUG   ] Got command RPT from MCS: ref #565548
2018-11-20 04:25:07 [DEBUG   ] summary = ERROR
2018-11-20 04:25:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565548  13 58442 15907553 A  ERRORERROR'
2018-11-20 04:25:07 [DEBUG   ] Got command RPT from MCS: ref #565549
2018-11-20 04:25:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:25:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565549  71 58442 15907757 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:25:07 [DEBUG   ] Got command RPT from MCS: ref #565550
2018-11-20 04:25:07 [DEBUG   ] summary = ERROR
2018-11-20 04:25:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565550  13 58442 15907949 A  ERRORERROR'
2018-11-20 04:25:08 [DEBUG   ] Got command RPT from MCS: ref #565551
2018-11-20 04:25:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:25:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565551  71 58442 15908153 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:25:08 [DEBUG   ] Got command RPT from MCS: ref #565552
2018-11-20 04:25:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:25:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565552  12 58442 15908361 A  ERRORB£ff'
2018-11-20 04:25:22 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 04:25:42 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 04:25:46 [DEBUG   ] Got command RPT from MCS: ref #565557
2018-11-20 04:25:46 [DEBUG   ] summary = ERROR
2018-11-20 04:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565557  13 58442 15946605 A  ERRORERROR'
2018-11-20 04:25:46 [DEBUG   ] Got command RPT from MCS: ref #565558
2018-11-20 04:25:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565558  71 58442 15946606 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:25:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:25:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366840552000000
2018-11-20 04:25:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:25:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:25:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:25:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:25:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366840552000000
2018-11-20 04:25:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:26:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:26:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:26:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:26:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:26:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:26:00 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:26:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:26:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:26:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:26:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:26:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:26:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:26:00 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:26:00 [DEBUG   ] Finished updating board flags in 1.417 seconds
2018-11-20 04:26:03 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 04:26:23 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 04:26:43 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 04:26:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:26:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366852312000000
2018-11-20 04:26:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:26:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:26:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:26:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:26:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366852312000000
2018-11-20 04:26:59 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 04:26:59 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:27:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:27:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:27:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:27:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:27:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:27:00 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:27:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:27:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:27:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:27:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:27:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:27:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:27:00 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:27:00 [DEBUG   ] Finished updating board flags in 1.388 seconds
2018-11-20 04:27:03 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 04:27:23 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 04:27:43 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:27:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:27:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366864072000000
2018-11-20 04:27:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:27:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:27:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:27:59 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:27:59 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366864072000000
2018-11-20 04:28:00 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:28:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:28:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302366864268000000
2018-11-20 04:28:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:28:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:28:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:28:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:28:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302366864268000000
2018-11-20 04:28:00 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:28:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:28:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:28:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:28:00 [WARNING ] (13, 1, 'bot')
2018-11-20 04:28:00 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:28:00 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 04:28:00 [DEBUG   ] Finished updating board flags in 1.360 seconds
2018-11-20 04:28:03 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:28:23 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:28:43 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 04:29:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:29:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:29:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:29:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:29:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:29:00 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:29:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 04:29:00 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:29:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:29:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:29:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:29:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:29:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:29:01 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:29:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:29:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:29:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:29:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:29:01 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:29:01 [DEBUG   ] Finished updating board flags in 2.063 seconds
2018-11-20 04:29:03 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:29:23 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 04:29:43 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:30:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:30:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:30:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:30:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:30:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:30:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 04:30:00 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:30:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:30:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:30:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:30:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:30:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:30:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 04:30:01 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:30:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:30:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:30:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:30:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:30:01 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:30:01 [DEBUG   ] Finished updating board flags in 1.993 seconds
2018-11-20 04:30:03 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:30:07 [DEBUG   ] Got command RPT from MCS: ref #565580
2018-11-20 04:30:07 [DEBUG   ] summary = ERROR
2018-11-20 04:30:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565580  13 58442 16207633 A  ERRORERROR'
2018-11-20 04:30:07 [DEBUG   ] Got command RPT from MCS: ref #565581
2018-11-20 04:30:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:30:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565581  98 58442 16207829 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:30:08 [DEBUG   ] Got command RPT from MCS: ref #565582
2018-11-20 04:30:08 [DEBUG   ] summary = ERROR
2018-11-20 04:30:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565582  13 58442 16208033 A  ERRORERROR'
2018-11-20 04:30:08 [DEBUG   ] Got command RPT from MCS: ref #565584
2018-11-20 04:30:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:30:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565584  98 58442 16208237 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:30:08 [DEBUG   ] Got command RPT from MCS: ref #565586
2018-11-20 04:30:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:30:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565586  12 58442 16208429 A  ERRORB¡ff'
2018-11-20 04:30:23 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:30:43 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 04:30:46 [DEBUG   ] Got command RPT from MCS: ref #565590
2018-11-20 04:30:46 [DEBUG   ] summary = ERROR
2018-11-20 04:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565590  13 58442 16246605 A  ERRORERROR'
2018-11-20 04:30:46 [DEBUG   ] Got command RPT from MCS: ref #565591
2018-11-20 04:30:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565591  98 58442 16246606 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:31:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:31:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:31:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:31:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:31:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:31:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:31:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 04:31:00 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:31:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:31:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:31:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:31:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:31:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:31:01 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:31:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:31:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:31:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:31:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:31:01 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:31:01 [DEBUG   ] Finished updating board flags in 1.808 seconds
2018-11-20 04:31:03 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:31:23 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 04:31:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 04:32:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:32:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:32:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:32:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:32:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:32:00 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:32:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:32:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:32:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:32:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:32:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:32:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 04:32:01 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:32:01 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:32:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:32:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:32:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:32:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:32:01 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:32:01 [DEBUG   ] Finished updating board flags in 1.786 seconds
2018-11-20 04:32:03 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 04:32:23 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 04:32:43 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 04:33:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:33:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366923068000000
2018-11-20 04:33:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:33:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:33:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:33:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:33:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366923068000000
2018-11-20 04:33:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 04:33:00 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:33:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 04:33:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:33:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:33:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:33:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:33:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:33:01 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:33:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:33:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:33:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:33:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:33:01 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:33:01 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:33:01 [DEBUG   ] Finished updating board flags in 1.623 seconds
2018-11-20 04:33:03 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 04:33:24 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 04:33:44 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 04:34:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:34:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366934828000000
2018-11-20 04:34:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:34:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:34:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:34:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:34:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366934828000000
2018-11-20 04:34:00 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:34:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:34:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:34:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:34:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:34:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:34:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa20, chip 4
2018-11-20 04:34:01 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:34:01 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:34:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:34:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:34:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:34:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:34:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:34:01 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:34:01 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:34:01 [DEBUG   ] Finished updating board flags in 1.584 seconds
2018-11-20 04:34:04 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 04:34:24 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 04:34:44 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 04:35:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:35:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366946588000000
2018-11-20 04:35:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:35:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:35:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:35:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:35:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366946588000000
2018-11-20 04:35:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:35:00 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:35:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 04:35:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:35:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:35:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:35:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:35:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:35:01 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:35:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:35:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:35:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:35:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:35:01 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:35:01 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:35:01 [DEBUG   ] Finished updating board flags in 1.423 seconds
2018-11-20 04:35:04 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 04:35:07 [DEBUG   ] Got command RPT from MCS: ref #565615
2018-11-20 04:35:07 [DEBUG   ] summary = ERROR
2018-11-20 04:35:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565615  13 58442 16507717 A  ERRORERROR'
2018-11-20 04:35:07 [DEBUG   ] Got command RPT from MCS: ref #565616
2018-11-20 04:35:07 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:35:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565616  71 58442 16507909 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:35:08 [DEBUG   ] Got command RPT from MCS: ref #565617
2018-11-20 04:35:08 [DEBUG   ] summary = ERROR
2018-11-20 04:35:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565617  13 58442 16508113 A  ERRORERROR'
2018-11-20 04:35:08 [DEBUG   ] Got command RPT from MCS: ref #565618
2018-11-20 04:35:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:35:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565618  71 58442 16508317 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:35:08 [DEBUG   ] Got command RPT from MCS: ref #565619
2018-11-20 04:35:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:35:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565619  12 58442 16508521 A  ERRORB£ff'
2018-11-20 04:35:24 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:35:44 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:35:46 [DEBUG   ] Got command RPT from MCS: ref #565620
2018-11-20 04:35:46 [DEBUG   ] summary = ERROR
2018-11-20 04:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565620  13 58442 16546601 A  ERRORERROR'
2018-11-20 04:35:46 [DEBUG   ] Got command RPT from MCS: ref #565621
2018-11-20 04:35:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565621  71 58442 16546602 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:36:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:36:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302366958348000000
2018-11-20 04:36:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:36:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:36:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:36:00 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:36:00 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302366958348000000
2018-11-20 04:36:00 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:36:00 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 04:36:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:36:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:36:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:36:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:36:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:36:01 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:36:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:36:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:36:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:36:01 [WARNING ] (13, 1, 'bot')
2018-11-20 04:36:01 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:36:01 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:36:01 [DEBUG   ] Finished updating board flags in 1.394 seconds
2018-11-20 04:36:04 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:36:24 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:36:44 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:37:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:37:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:37:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:37:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:37:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:37:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:37:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 04:37:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 04:37:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:37:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:37:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:37:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:37:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:37:02 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:37:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:37:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:37:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:37:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:37:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:37:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:37:02 [DEBUG   ] Finished updating board flags in 2.235 seconds
2018-11-20 04:37:04 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:37:24 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:37:44 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:38:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:38:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:38:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:38:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:38:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:38:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:38:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:38:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:38:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:38:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:38:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:38:02 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 04:38:02 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:38:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:38:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:38:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:38:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:38:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:38:02 [DEBUG   ] Finished updating board flags in 2.270 seconds
2018-11-20 04:38:04 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:38:24 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 04:38:44 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:39:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:39:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:39:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:39:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:39:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:39:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 04:39:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:39:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:39:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:39:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:39:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:39:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:39:02 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:39:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 04:39:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:39:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 04:39:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:39:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:39:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:39:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:39:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:39:02 [DEBUG   ] Finished updating board flags in 2.113 seconds
2018-11-20 04:39:04 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:39:24 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 04:39:44 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:40:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:40:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:40:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:40:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:40:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:40:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:40:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:40:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:40:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:40:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:40:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:40:02 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:40:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:40:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:40:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:40:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:40:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:40:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:40:02 [DEBUG   ] Finished updating board flags in 1.997 seconds
2018-11-20 04:40:04 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:40:07 [DEBUG   ] Got command RPT from MCS: ref #565643
2018-11-20 04:40:07 [DEBUG   ] summary = ERROR
2018-11-20 04:40:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565643  13 58442 16807793 A  ERRORERROR'
2018-11-20 04:40:07 [DEBUG   ] Got command RPT from MCS: ref #565644
2018-11-20 04:40:07 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:40:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565644  98 58442 16807997 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:40:08 [DEBUG   ] Got command RPT from MCS: ref #565645
2018-11-20 04:40:08 [DEBUG   ] summary = ERROR
2018-11-20 04:40:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565645  13 58442 16808201 A  ERRORERROR'
2018-11-20 04:40:08 [DEBUG   ] Got command RPT from MCS: ref #565647
2018-11-20 04:40:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:40:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565647  98 58442 16808393 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:40:08 [DEBUG   ] Got command RPT from MCS: ref #565648
2018-11-20 04:40:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:40:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565648  12 58442 16808597 A  ERRORB£ff'
2018-11-20 04:40:24 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:40:44 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:40:46 [DEBUG   ] Got command RPT from MCS: ref #565657
2018-11-20 04:40:46 [DEBUG   ] summary = ERROR
2018-11-20 04:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565657  13 58442 16846601 A  ERRORERROR'
2018-11-20 04:40:46 [DEBUG   ] Got command RPT from MCS: ref #565658
2018-11-20 04:40:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565658  98 58442 16846602 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:41:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:41:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:41:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:41:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:41:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:41:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:41:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 04:41:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 04:41:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:41:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:41:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:41:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:41:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:41:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:41:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 04:41:02 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:41:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:41:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:41:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:41:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:41:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:41:02 [DEBUG   ] Finished updating board flags in 1.817 seconds
2018-11-20 04:41:04 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:41:24 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:41:44 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 04:42:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:42:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:42:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:42:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:42:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:42:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 04:42:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:42:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:42:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:42:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:42:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:42:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:42:02 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:42:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:42:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:42:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:42:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:42:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:42:02 [DEBUG   ] Finished updating board flags in 1.778 seconds
2018-11-20 04:42:04 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:42:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 04:42:44 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:43:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:43:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367040864000000
2018-11-20 04:43:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:43:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:43:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:43:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:43:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367040864000000
2018-11-20 04:43:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 04:43:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:43:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:43:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:43:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:43:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:43:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:43:02 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:43:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:43:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:43:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:43:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:43:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:43:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:43:02 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:43:02 [DEBUG   ] Finished updating board flags in 1.618 seconds
2018-11-20 04:43:04 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:43:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:43:44 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 04:44:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:44:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367052624000000
2018-11-20 04:44:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:44:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:44:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:44:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:44:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367052624000000
2018-11-20 04:44:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:44:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:44:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:44:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:44:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:44:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:44:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 04:44:02 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:44:02 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 04:44:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:44:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:44:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:44:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:44:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:44:02 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:44:02 [DEBUG   ] Finished updating board flags in 1.639 seconds
2018-11-20 04:44:04 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:44:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:44:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:45:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:45:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367064384000000
2018-11-20 04:45:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:45:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:45:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:45:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:45:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367064384000000
2018-11-20 04:45:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:45:01 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 04:45:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:45:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:45:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:45:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:45:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:45:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:45:02 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:45:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:45:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:45:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:45:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:45:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:45:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:45:02 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:45:02 [DEBUG   ] Finished updating board flags in 1.676 seconds
2018-11-20 04:45:04 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 04:45:07 [DEBUG   ] Got command RPT from MCS: ref #565678
2018-11-20 04:45:07 [DEBUG   ] summary = ERROR
2018-11-20 04:45:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565678  13 58442 17107881 A  ERRORERROR'
2018-11-20 04:45:08 [DEBUG   ] Got command RPT from MCS: ref #565679
2018-11-20 04:45:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:45:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565679  71 58442 17108089 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:45:08 [DEBUG   ] Got command RPT from MCS: ref #565680
2018-11-20 04:45:08 [DEBUG   ] summary = ERROR
2018-11-20 04:45:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565680  13 58442 17108281 A  ERRORERROR'
2018-11-20 04:45:08 [DEBUG   ] Got command RPT from MCS: ref #565681
2018-11-20 04:45:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:45:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565681  71 58442 17108473 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:45:08 [DEBUG   ] Got command RPT from MCS: ref #565682
2018-11-20 04:45:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:45:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565682  12 58442 17108677 A  ERRORB¡ff'
2018-11-20 04:45:24 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:45:44 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 04:45:46 [DEBUG   ] Got command RPT from MCS: ref #565683
2018-11-20 04:45:46 [DEBUG   ] summary = ERROR
2018-11-20 04:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565683  13 58442 17146601 A  ERRORERROR'
2018-11-20 04:45:46 [DEBUG   ] Got command RPT from MCS: ref #565684
2018-11-20 04:45:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565684  71 58442 17146602 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:46:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:46:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367076144000000
2018-11-20 04:46:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:46:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:46:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:46:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:46:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367076144000000
2018-11-20 04:46:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:46:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:46:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:46:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:46:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:46:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:46:02 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:46:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 04:46:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:46:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:46:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:46:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:46:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:46:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:46:02 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:46:02 [DEBUG   ] Finished updating board flags in 1.478 seconds
2018-11-20 04:46:04 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 04:46:24 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 04:46:44 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 04:47:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:47:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367087904000000
2018-11-20 04:47:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:47:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:47:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:47:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:47:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367087904000000
2018-11-20 04:47:01 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:47:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:47:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367088100000000
2018-11-20 04:47:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:47:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:47:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:47:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:47:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367088100000000
2018-11-20 04:47:02 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 04:47:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 04:47:02 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:47:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:47:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:47:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:47:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:47:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:47:02 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 04:47:02 [DEBUG   ] Finished updating board flags in 1.490 seconds
2018-11-20 04:47:05 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 04:47:25 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 04:47:45 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 04:48:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:48:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367099664000000
2018-11-20 04:48:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:48:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:48:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:48:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:48:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367099664000000
2018-11-20 04:48:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:48:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 5
2018-11-20 04:48:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:48:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367099860000000
2018-11-20 04:48:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:48:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:48:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:48:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:48:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367099860000000
2018-11-20 04:48:02 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 04:48:02 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:48:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:48:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:48:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:48:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:48:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:48:02 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 04:48:02 [DEBUG   ] Finished updating board flags in 1.483 seconds
2018-11-20 04:48:05 [DEBUG   ] Finished updating ant stats in 1.191 seconds
2018-11-20 04:48:25 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:48:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:49:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:49:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367111424000000
2018-11-20 04:49:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:49:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:49:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:49:01 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:49:01 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367111424000000
2018-11-20 04:49:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:49:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:49:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367111620000000
2018-11-20 04:49:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:49:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:49:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:49:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:49:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367111620000000
2018-11-20 04:49:02 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:49:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:49:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:49:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:49:02 [WARNING ] (13, 1, 'bot')
2018-11-20 04:49:02 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:49:02 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 04:49:02 [DEBUG   ] Finished updating board flags in 1.494 seconds
2018-11-20 04:49:05 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:49:25 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:49:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:50:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:50:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:50:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:50:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:50:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:50:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:50:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:50:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:50:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:50:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:50:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:50:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 04:50:03 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:50:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 04:50:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:50:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:50:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:50:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:50:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:50:03 [DEBUG   ] Finished updating board flags in 2.166 seconds
2018-11-20 04:50:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:50:07 [DEBUG   ] Got command RPT from MCS: ref #565710
2018-11-20 04:50:07 [DEBUG   ] summary = ERROR
2018-11-20 04:50:07 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565710  13 58442 17407953 A  ERRORERROR'
2018-11-20 04:50:08 [DEBUG   ] Got command RPT from MCS: ref #565711
2018-11-20 04:50:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:50:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565711  98 58442 17408157 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:50:08 [DEBUG   ] Got command RPT from MCS: ref #565712
2018-11-20 04:50:08 [DEBUG   ] summary = ERROR
2018-11-20 04:50:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565712  13 58442 17408362 A  ERRORERROR'
2018-11-20 04:50:08 [DEBUG   ] Got command RPT from MCS: ref #565714
2018-11-20 04:50:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:50:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565714  98 58442 17408553 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:50:08 [DEBUG   ] Got command RPT from MCS: ref #565715
2018-11-20 04:50:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:50:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565715  12 58442 17408757 A  ERRORB¡ff'
2018-11-20 04:50:25 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:50:45 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 04:50:46 [DEBUG   ] Got command RPT from MCS: ref #565720
2018-11-20 04:50:46 [DEBUG   ] summary = ERROR
2018-11-20 04:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565720  13 58442 17446605 A  ERRORERROR'
2018-11-20 04:50:46 [DEBUG   ] Got command RPT from MCS: ref #565721
2018-11-20 04:50:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 04:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565721  98 58442 17446606 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 04:51:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:51:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:51:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:51:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:51:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:51:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 04:51:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:51:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:51:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:51:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:51:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:51:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:51:03 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:51:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:51:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:51:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:51:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:51:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:51:03 [DEBUG   ] Finished updating board flags in 1.972 seconds
2018-11-20 04:51:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:51:25 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:51:45 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:52:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:52:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:52:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:52:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:52:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:52:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:52:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:52:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:52:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:52:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:52:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:52:03 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:52:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:52:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:52:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:52:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:52:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:52:03 [DEBUG   ] Finished updating board flags in 1.830 seconds
2018-11-20 04:52:05 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 04:52:25 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 04:52:45 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 04:53:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:53:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:53:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:53:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:53:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:53:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:53:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:53:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:53:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:53:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:53:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:53:03 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:53:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:53:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:53:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:53:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:53:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:53:03 [DEBUG   ] Finished updating board flags in 1.770 seconds
2018-11-20 04:53:05 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 04:53:26 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 04:53:46 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 04:54:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:54:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367170420000000
2018-11-20 04:54:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:54:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:54:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:54:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:54:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367170420000000
2018-11-20 04:54:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 04:54:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 04:54:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:54:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:54:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:54:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:54:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:54:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:54:03 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:54:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:54:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:54:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:54:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:54:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:54:03 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:54:03 [DEBUG   ] Finished updating board flags in 1.617 seconds
2018-11-20 04:54:06 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:54:26 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 04:54:46 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:55:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:55:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367182180000000
2018-11-20 04:55:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:55:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:55:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:55:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:55:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367182180000000
2018-11-20 04:55:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:55:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:55:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 04:55:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:55:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:55:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:55:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:55:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:55:03 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:55:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:55:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 04:55:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:55:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:55:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:55:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:55:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:55:03 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:55:03 [DEBUG   ] Finished updating board flags in 1.600 seconds
2018-11-20 04:55:06 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 04:55:08 [DEBUG   ] Got command RPT from MCS: ref #565741
2018-11-20 04:55:08 [DEBUG   ] summary = ERROR
2018-11-20 04:55:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565741  13 58442 17708041 A  ERRORERROR'
2018-11-20 04:55:08 [DEBUG   ] Got command RPT from MCS: ref #565742
2018-11-20 04:55:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:55:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565742  71 58442 17708233 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:55:08 [DEBUG   ] Got command RPT from MCS: ref #565743
2018-11-20 04:55:08 [DEBUG   ] summary = ERROR
2018-11-20 04:55:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565743  13 58442 17708437 A  ERRORERROR'
2018-11-20 04:55:08 [DEBUG   ] Got command RPT from MCS: ref #565744
2018-11-20 04:55:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:55:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565744  71 58442 17708641 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:55:08 [DEBUG   ] Got command RPT from MCS: ref #565745
2018-11-20 04:55:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 04:55:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565745  12 58442 17708833 A  ERRORB£ff'
2018-11-20 04:55:26 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 04:55:46 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 04:55:46 [DEBUG   ] Got command RPT from MCS: ref #565750
2018-11-20 04:55:46 [DEBUG   ] summary = ERROR
2018-11-20 04:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565750  13 58442 17746601 A  ERRORERROR'
2018-11-20 04:55:46 [DEBUG   ] Got command RPT from MCS: ref #565751
2018-11-20 04:55:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 04:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565751  71 58442 17746602 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 04:56:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:56:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367193940000000
2018-11-20 04:56:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:56:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:56:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:56:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:56:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367193940000000
2018-11-20 04:56:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 04:56:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:56:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:56:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:56:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:56:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:56:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:56:03 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:56:03 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:56:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:56:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:56:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:56:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:56:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:56:03 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:56:03 [DEBUG   ] Finished updating board flags in 1.406 seconds
2018-11-20 04:56:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:56:26 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 04:56:46 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:57:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:57:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367205700000000
2018-11-20 04:57:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:57:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:57:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:57:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:57:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367205700000000
2018-11-20 04:57:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 04:57:02 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 04:57:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:57:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:57:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:57:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:57:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:57:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:57:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 04:57:03 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:57:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:57:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:57:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:57:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:57:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:57:03 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:57:03 [DEBUG   ] Finished updating board flags in 1.400 seconds
2018-11-20 04:57:06 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 04:57:26 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 04:57:46 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 04:58:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:58:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367217460000000
2018-11-20 04:58:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:58:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:58:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:58:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:58:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367217460000000
2018-11-20 04:58:02 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:58:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:58:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:58:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:58:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:58:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:58:03 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 04:58:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 04:58:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:58:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:58:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:58:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:58:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:58:03 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 04:58:03 [DEBUG   ] Finished updating board flags in 1.508 seconds
2018-11-20 04:58:06 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 04:58:26 [DEBUG   ] Finished updating ant stats in 0.797 seconds
2018-11-20 04:58:46 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 04:59:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 04:59:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367229220000000
2018-11-20 04:59:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 04:59:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 04:59:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 04:59:02 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 04:59:02 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367229220000000
2018-11-20 04:59:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 04:59:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 04:59:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367229416000000
2018-11-20 04:59:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 04:59:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 04:59:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 04:59:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 04:59:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367229416000000
2018-11-20 04:59:03 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 04:59:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:59:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:59:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:59:03 [WARNING ] (13, 1, 'bot')
2018-11-20 04:59:03 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 04:59:03 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 04:59:03 [DEBUG   ] Finished updating board flags in 1.524 seconds
2018-11-20 04:59:06 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 04:59:26 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 04:59:46 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 05:00:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:00:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:00:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:00:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:00:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:00:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 05:00:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:00:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:00:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:00:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:00:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:00:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:00:04 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:00:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:00:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:00:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:00:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:00:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:00:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:00:04 [DEBUG   ] Finished updating board flags in 2.290 seconds
2018-11-20 05:00:06 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:00:08 [DEBUG   ] Got command RPT from MCS: ref #565773
2018-11-20 05:00:08 [DEBUG   ] summary = ERROR
2018-11-20 05:00:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565773  13 58442 18008109 A  ERRORERROR'
2018-11-20 05:00:08 [DEBUG   ] Got command RPT from MCS: ref #565774
2018-11-20 05:00:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:00:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565774  98 58442 18008313 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:00:08 [DEBUG   ] Got command RPT from MCS: ref #565775
2018-11-20 05:00:08 [DEBUG   ] summary = ERROR
2018-11-20 05:00:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565775  13 58442 18008517 A  ERRORERROR'
2018-11-20 05:00:08 [DEBUG   ] Got command RPT from MCS: ref #565777
2018-11-20 05:00:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:00:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565777  98 58442 18008721 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:00:08 [DEBUG   ] Got command RPT from MCS: ref #565778
2018-11-20 05:00:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:00:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565778  12 58442 18008913 A  ERRORB¢ff'
2018-11-20 05:00:26 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 05:00:46 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:00:46 [DEBUG   ] Got command RPT from MCS: ref #565783
2018-11-20 05:00:46 [DEBUG   ] summary = ERROR
2018-11-20 05:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565783  13 58442 18046609 A  ERRORERROR'
2018-11-20 05:00:46 [DEBUG   ] Got command RPT from MCS: ref #565784
2018-11-20 05:00:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565784  98 58442 18046610 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:01:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:01:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:01:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:01:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:01:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:01:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 05:01:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:01:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:01:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:01:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:01:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:01:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:01:04 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:01:04 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:01:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:01:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:01:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:01:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:01:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:01:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:01:04 [DEBUG   ] Finished updating board flags in 2.076 seconds
2018-11-20 05:01:06 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 05:01:26 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 05:01:46 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 05:02:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:02:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:02:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:02:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:02:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:02:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 05:02:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:02:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:02:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:02:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:02:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:02:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:02:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:02:04 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:02:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 05:02:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:02:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:02:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:02:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:02:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:02:04 [DEBUG   ] Finished updating board flags in 2.027 seconds
2018-11-20 05:02:07 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 05:02:27 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 05:02:47 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 05:03:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:03:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:03:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:03:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:03:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:03:03 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:03:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 05:03:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 05:03:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:03:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:03:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:03:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:03:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:03:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:03:04 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:03:04 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:03:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:03:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:03:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:03:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:03:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:03:04 [DEBUG   ] Finished updating board flags in 1.985 seconds
2018-11-20 05:03:07 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 05:03:27 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 05:03:47 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 05:04:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:04:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:04:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:04:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:04:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:04:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 05:04:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:04:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:04:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:04:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:04:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:04:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:04:04 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:04:04 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:04:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 05:04:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:04:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:04:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:04:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:04:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:04:04 [DEBUG   ] Finished updating board flags in 1.811 seconds
2018-11-20 05:04:07 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 05:04:27 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 05:04:47 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:05:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:05:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:05:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:05:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:05:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:05:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 05:05:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 05:05:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:05:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:05:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:05:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:05:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:05:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:05:04 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:05:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:05:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:05:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:05:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:05:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:05:04 [DEBUG   ] Finished updating board flags in 1.785 seconds
2018-11-20 05:05:07 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 05:05:08 [DEBUG   ] Got command RPT from MCS: ref #565808
2018-11-20 05:05:08 [DEBUG   ] summary = ERROR
2018-11-20 05:05:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565808  13 58442 18308189 A  ERRORERROR'
2018-11-20 05:05:08 [DEBUG   ] Got command RPT from MCS: ref #565809
2018-11-20 05:05:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:05:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565809  98 58442 18308393 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:05:08 [DEBUG   ] Got command RPT from MCS: ref #565810
2018-11-20 05:05:08 [DEBUG   ] summary = ERROR
2018-11-20 05:05:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565810  13 58442 18308597 A  ERRORERROR'
2018-11-20 05:05:08 [DEBUG   ] Got command RPT from MCS: ref #565811
2018-11-20 05:05:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:05:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565811  98 58442 18308801 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:05:08 [DEBUG   ] Got command RPT from MCS: ref #565812
2018-11-20 05:05:08 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:05:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565812  12 58442 18308993 A  ERRORB¡ff'
2018-11-20 05:05:27 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:05:46 [DEBUG   ] Got command RPT from MCS: ref #565813
2018-11-20 05:05:46 [DEBUG   ] summary = ERROR
2018-11-20 05:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565813  13 58442 18346605 A  ERRORERROR'
2018-11-20 05:05:46 [DEBUG   ] Got command RPT from MCS: ref #565814
2018-11-20 05:05:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565814  98 58442 18346606 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:05:47 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 05:06:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:06:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367311736000000
2018-11-20 05:06:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:06:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:06:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:06:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:06:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367311736000000
2018-11-20 05:06:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:06:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:06:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:06:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:06:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:06:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:06:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:06:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 05:06:04 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:06:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:06:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:06:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:06:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:06:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:06:04 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:06:04 [DEBUG   ] Finished updating board flags in 1.614 seconds
2018-11-20 05:06:07 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:06:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:06:47 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:07:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:07:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367323496000000
2018-11-20 05:07:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:07:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:07:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:07:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:07:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367323496000000
2018-11-20 05:07:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:07:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:07:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:07:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:07:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:07:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:07:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:07:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:07:04 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:07:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:07:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:07:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:07:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:07:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:07:04 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:07:04 [DEBUG   ] Finished updating board flags in 1.590 seconds
2018-11-20 05:07:07 [DEBUG   ] Finished updating ant stats in 0.595 seconds
2018-11-20 05:07:28 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 05:07:48 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 05:08:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:08:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367335256000000
2018-11-20 05:08:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:08:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:08:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:08:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:08:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367335256000000
2018-11-20 05:08:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:08:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:08:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:08:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:08:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:08:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:08:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:08:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:08:04 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:08:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:08:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:08:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:08:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:08:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:08:04 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:08:04 [DEBUG   ] Finished updating board flags in 1.414 seconds
2018-11-20 05:08:08 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 05:08:28 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 05:08:48 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 05:09:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:09:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367347016000000
2018-11-20 05:09:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:09:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:09:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:09:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:09:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367347016000000
2018-11-20 05:09:03 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:09:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:09:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:09:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:09:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:09:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:09:04 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:09:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:09:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:09:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:09:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:09:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:09:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:09:04 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:09:04 [DEBUG   ] Finished updating board flags in 1.387 seconds
2018-11-20 05:09:08 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 05:09:28 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 05:09:48 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 05:10:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:10:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367358776000000
2018-11-20 05:10:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:10:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:10:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:10:03 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:10:03 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367358776000000
2018-11-20 05:10:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 05:10:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 05:10:03 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 05:10:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:10:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:10:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367358972000000
2018-11-20 05:10:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:10:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:10:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:10:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:10:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367358972000000
2018-11-20 05:10:04 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:10:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:10:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:10:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:10:04 [WARNING ] (13, 1, 'bot')
2018-11-20 05:10:04 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:10:04 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 05:10:04 [DEBUG   ] Finished updating board flags in 1.359 seconds
2018-11-20 05:10:08 [DEBUG   ] Got command RPT from MCS: ref #565836
2018-11-20 05:10:08 [DEBUG   ] summary = ERROR
2018-11-20 05:10:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565836  13 58442 18608273 A  ERRORERROR'
2018-11-20 05:10:08 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 05:10:08 [DEBUG   ] Got command RPT from MCS: ref #565837
2018-11-20 05:10:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 05:10:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565837  79 58442 18608477 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 05:10:08 [DEBUG   ] Got command RPT from MCS: ref #565838
2018-11-20 05:10:08 [DEBUG   ] summary = ERROR
2018-11-20 05:10:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565838  13 58442 18608669 A  ERRORERROR'
2018-11-20 05:10:08 [DEBUG   ] Got command RPT from MCS: ref #565840
2018-11-20 05:10:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 05:10:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565840  79 58442 18608873 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 05:10:09 [DEBUG   ] Got command RPT from MCS: ref #565842
2018-11-20 05:10:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:10:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565842  12 58442 18609078 A  ERRORB¢ff'
2018-11-20 05:10:28 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 05:10:46 [DEBUG   ] Got command RPT from MCS: ref #565850
2018-11-20 05:10:46 [DEBUG   ] summary = ERROR
2018-11-20 05:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565850  13 58442 18646605 A  ERRORERROR'
2018-11-20 05:10:46 [DEBUG   ] Got command RPT from MCS: ref #565851
2018-11-20 05:10:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 05:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565851  79 58442 18646606 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 05:10:48 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 05:11:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:11:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:11:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:11:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:11:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:11:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 05:11:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:11:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 05:11:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:11:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:11:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:11:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:11:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:11:05 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:11:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:11:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:11:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:11:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:11:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:11:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:11:05 [DEBUG   ] Finished updating board flags in 2.059 seconds
2018-11-20 05:11:08 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:11:28 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 05:11:48 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:12:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:12:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:12:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:12:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:12:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:12:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:12:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 05:12:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:12:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:12:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:12:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:12:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:12:05 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:12:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:12:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:12:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:12:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:12:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:12:05 [DEBUG   ] Finished updating board flags in 2.009 seconds
2018-11-20 05:12:08 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:12:28 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:12:48 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:13:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:13:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:13:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:13:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:13:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:13:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:13:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:13:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:13:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:13:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:13:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:13:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:13:05 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:13:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:13:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:13:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:13:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:13:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:13:05 [DEBUG   ] Finished updating board flags in 1.994 seconds
2018-11-20 05:13:08 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:13:28 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 05:13:48 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:14:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:14:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:14:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:14:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:14:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:14:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:14:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:14:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:14:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:14:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:14:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:14:05 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:14:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:14:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:14:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:14:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:14:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:14:05 [DEBUG   ] Finished updating board flags in 1.807 seconds
2018-11-20 05:14:08 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 05:14:28 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:14:48 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 05:15:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:15:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:15:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:15:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:15:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:15:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:15:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:15:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:15:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:15:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:15:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:15:05 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:15:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:15:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:15:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:15:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:15:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:15:05 [DEBUG   ] Finished updating board flags in 1.875 seconds
2018-11-20 05:15:08 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 05:15:08 [DEBUG   ] Got command RPT from MCS: ref #565871
2018-11-20 05:15:08 [DEBUG   ] summary = ERROR
2018-11-20 05:15:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565871  13 58442 18908349 A  ERRORERROR'
2018-11-20 05:15:08 [DEBUG   ] Got command RPT from MCS: ref #565872
2018-11-20 05:15:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:15:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565872  98 58442 18908553 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:15:08 [DEBUG   ] Got command RPT from MCS: ref #565873
2018-11-20 05:15:08 [DEBUG   ] summary = ERROR
2018-11-20 05:15:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565873  13 58442 18908757 A  ERRORERROR'
2018-11-20 05:15:08 [DEBUG   ] Got command RPT from MCS: ref #565874
2018-11-20 05:15:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:15:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565874  98 58442 18908949 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:15:09 [DEBUG   ] Got command RPT from MCS: ref #565875
2018-11-20 05:15:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:15:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565875  12 58442 18909153 A  ERRORB£ff'
2018-11-20 05:15:28 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:15:46 [DEBUG   ] Got command RPT from MCS: ref #565876
2018-11-20 05:15:46 [DEBUG   ] summary = ERROR
2018-11-20 05:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565876  13 58442 18946609 A  ERRORERROR'
2018-11-20 05:15:46 [DEBUG   ] Got command RPT from MCS: ref #565877
2018-11-20 05:15:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565877  98 58442 18946610 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:15:48 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:16:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:16:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:16:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:16:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:16:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:16:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:16:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:16:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:16:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:16:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:16:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:16:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:16:05 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:16:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:16:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:16:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:16:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:16:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:16:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:16:05 [DEBUG   ] Finished updating board flags in 1.821 seconds
2018-11-20 05:16:08 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 05:16:28 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 05:16:48 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 05:17:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:17:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:17:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:17:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:17:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:17:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:17:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 05:17:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:17:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:17:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:17:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:17:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:17:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:17:05 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:17:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:17:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:17:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:17:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:17:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:17:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:17:05 [DEBUG   ] Finished updating board flags in 1.699 seconds
2018-11-20 05:17:09 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 05:17:29 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 05:17:49 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 05:18:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:18:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367453052000000
2018-11-20 05:18:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:18:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:18:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:18:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:18:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367453052000000
2018-11-20 05:18:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:18:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:18:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:18:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:18:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:18:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:18:05 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:18:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:18:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:18:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:18:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:18:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:18:05 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:18:05 [DEBUG   ] Finished updating board flags in 1.613 seconds
2018-11-20 05:18:09 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 05:18:29 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 05:18:49 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 05:19:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:19:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367464812000000
2018-11-20 05:19:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:19:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:19:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:19:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:19:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367464812000000
2018-11-20 05:19:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:19:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:19:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:19:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:19:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:19:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:19:05 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:19:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:19:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:19:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:19:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:19:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:19:05 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:19:05 [DEBUG   ] Finished updating board flags in 1.635 seconds
2018-11-20 05:19:09 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 05:19:29 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 05:19:49 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 05:20:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:20:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367476572000000
2018-11-20 05:20:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:20:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:20:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:20:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:20:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367476572000000
2018-11-20 05:20:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:20:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:20:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:20:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:20:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:20:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:20:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 05:20:05 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:20:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:20:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:20:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:20:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:20:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:20:05 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:20:05 [DEBUG   ] Finished updating board flags in 1.563 seconds
2018-11-20 05:20:08 [DEBUG   ] Got command RPT from MCS: ref #565903
2018-11-20 05:20:08 [DEBUG   ] summary = ERROR
2018-11-20 05:20:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565903  13 58442 19208437 A  ERRORERROR'
2018-11-20 05:20:08 [DEBUG   ] Got command RPT from MCS: ref #565904
2018-11-20 05:20:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 05:20:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565904  71 58442 19208629 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 05:20:08 [DEBUG   ] Got command RPT from MCS: ref #565905
2018-11-20 05:20:08 [DEBUG   ] summary = ERROR
2018-11-20 05:20:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565905  13 58442 19208837 A  ERRORERROR'
2018-11-20 05:20:09 [DEBUG   ] Got command RPT from MCS: ref #565907
2018-11-20 05:20:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 05:20:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565907  71 58442 19209029 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 05:20:09 [DEBUG   ] Got command RPT from MCS: ref #565908
2018-11-20 05:20:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:20:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565908  12 58442 19209233 A  ERRORB¡ff'
2018-11-20 05:20:09 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:20:29 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:20:46 [DEBUG   ] Got command RPT from MCS: ref #565913
2018-11-20 05:20:46 [DEBUG   ] summary = ERROR
2018-11-20 05:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565913  13 58442 19246609 A  ERRORERROR'
2018-11-20 05:20:46 [DEBUG   ] Got command RPT from MCS: ref #565914
2018-11-20 05:20:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 05:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565914  71 58442 19246610 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 05:20:49 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-20 05:21:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:21:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367488332000000
2018-11-20 05:21:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:21:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:21:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:21:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:21:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367488332000000
2018-11-20 05:21:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:21:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:21:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:21:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:21:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:21:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:21:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:21:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:21:05 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:21:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 05:21:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:21:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:21:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:21:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:21:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:21:05 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:21:05 [DEBUG   ] Finished updating board flags in 1.410 seconds
2018-11-20 05:21:09 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 05:21:29 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:21:49 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:22:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:22:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367500092000000
2018-11-20 05:22:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:22:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:22:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:22:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:22:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367500092000000
2018-11-20 05:22:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:22:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:22:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:22:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:22:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:22:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:22:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:22:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:22:05 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:22:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:22:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:22:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:22:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:22:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:22:05 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:22:05 [DEBUG   ] Finished updating board flags in 1.411 seconds
2018-11-20 05:22:09 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:22:29 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 05:22:50 [DEBUG   ] Finished updating ant stats in 1.403 seconds
2018-11-20 05:23:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:23:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367511852000000
2018-11-20 05:23:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:23:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:23:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:23:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:23:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367511852000000
2018-11-20 05:23:04 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:23:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:23:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:23:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:23:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:23:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:23:05 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:23:05 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:23:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:23:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:23:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:23:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:23:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:23:05 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:23:05 [DEBUG   ] Finished updating board flags in 1.383 seconds
2018-11-20 05:23:09 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 05:23:30 [DEBUG   ] Finished updating ant stats in 1.403 seconds
2018-11-20 05:23:49 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 05:24:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:24:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367523612000000
2018-11-20 05:24:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:24:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:24:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:24:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:24:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367523612000000
2018-11-20 05:24:04 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:24:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:24:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:24:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367523808000000
2018-11-20 05:24:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:24:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:24:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:24:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:24:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367523808000000
2018-11-20 05:24:05 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:24:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:24:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:24:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:24:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:24:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:24:05 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 05:24:05 [DEBUG   ] Finished updating board flags in 1.409 seconds
2018-11-20 05:24:10 [DEBUG   ] Finished updating ant stats in 1.404 seconds
2018-11-20 05:24:30 [DEBUG   ] Finished updating ant stats in 1.329 seconds
2018-11-20 05:24:50 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 05:25:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:25:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367535372000000
2018-11-20 05:25:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:25:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:25:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:25:04 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:25:04 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367535372000000
2018-11-20 05:25:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:25:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:25:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367535568000000
2018-11-20 05:25:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:25:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:25:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:25:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:25:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367535568000000
2018-11-20 05:25:05 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:25:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:25:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:25:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:25:05 [WARNING ] (13, 1, 'bot')
2018-11-20 05:25:05 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:25:05 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 05:25:05 [DEBUG   ] Finished updating board flags in 1.392 seconds
2018-11-20 05:25:08 [DEBUG   ] Got command RPT from MCS: ref #565934
2018-11-20 05:25:08 [DEBUG   ] summary = ERROR
2018-11-20 05:25:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565934  13 58442 19508509 A  ERRORERROR'
2018-11-20 05:25:08 [DEBUG   ] Got command RPT from MCS: ref #565935
2018-11-20 05:25:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 05:25:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565935  79 58442 19508713 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 05:25:08 [DEBUG   ] Got command RPT from MCS: ref #565936
2018-11-20 05:25:08 [DEBUG   ] summary = ERROR
2018-11-20 05:25:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565936  13 58442 19508909 A  ERRORERROR'
2018-11-20 05:25:09 [DEBUG   ] Got command RPT from MCS: ref #565937
2018-11-20 05:25:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 05:25:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565937  79 58442 19509117 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 05:25:09 [DEBUG   ] Got command RPT from MCS: ref #565938
2018-11-20 05:25:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:25:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565938  12 58442 19509322 A  ERRORB ff'
2018-11-20 05:25:10 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 05:25:30 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 05:25:46 [DEBUG   ] Got command RPT from MCS: ref #565943
2018-11-20 05:25:46 [DEBUG   ] summary = ERROR
2018-11-20 05:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565943  13 58442 19546617 A  ERRORERROR'
2018-11-20 05:25:46 [DEBUG   ] Got command RPT from MCS: ref #565944
2018-11-20 05:25:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 05:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565944  79 58442 19546618 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 05:25:50 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 05:26:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:26:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:26:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:26:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:26:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:26:05 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:26:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:26:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:26:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:26:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:26:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:26:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:26:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 05:26:06 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:26:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:26:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:26:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:26:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:26:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:26:06 [DEBUG   ] Finished updating board flags in 2.029 seconds
2018-11-20 05:26:10 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 05:26:30 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:26:50 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:27:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:27:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:27:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:27:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:27:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:27:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:27:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:27:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:27:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:27:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:27:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:27:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:27:06 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:27:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:27:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:27:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:27:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:27:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:27:06 [DEBUG   ] Finished updating board flags in 1.980 seconds
2018-11-20 05:27:10 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:27:30 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 05:27:50 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 05:28:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:28:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:28:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:28:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:28:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:28:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:28:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:28:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:28:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:28:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:28:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:28:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:28:06 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:28:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:28:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:28:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:28:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:28:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:28:06 [DEBUG   ] Finished updating board flags in 1.811 seconds
2018-11-20 05:28:10 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:28:30 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:28:50 [DEBUG   ] Finished updating ant stats in 0.597 seconds
2018-11-20 05:29:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:29:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:29:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:29:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:29:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:29:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:29:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:29:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:29:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:29:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:29:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:29:06 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:29:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:29:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:29:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:29:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:29:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:29:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:29:06 [DEBUG   ] Finished updating board flags in 1.856 seconds
2018-11-20 05:29:10 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 05:29:30 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 05:29:51 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 05:30:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:30:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:30:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:30:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:30:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:30:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 05:30:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:30:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:30:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:30:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:30:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:30:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:30:06 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:30:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:30:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:30:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:30:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:30:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:30:06 [DEBUG   ] Finished updating board flags in 1.741 seconds
2018-11-20 05:30:08 [DEBUG   ] Got command RPT from MCS: ref #565966
2018-11-20 05:30:08 [DEBUG   ] summary = ERROR
2018-11-20 05:30:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565966  13 58442 19808593 A  ERRORERROR'
2018-11-20 05:30:08 [DEBUG   ] Got command RPT from MCS: ref #565967
2018-11-20 05:30:08 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:30:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565967  98 58442 19808797 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:30:08 [DEBUG   ] Got command RPT from MCS: ref #565968
2018-11-20 05:30:08 [DEBUG   ] summary = ERROR
2018-11-20 05:30:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565968  13 58442 19808989 A  ERRORERROR'
2018-11-20 05:30:09 [DEBUG   ] Got command RPT from MCS: ref #565970
2018-11-20 05:30:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:30:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565970  98 58442 19809193 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:30:09 [DEBUG   ] Got command RPT from MCS: ref #565972
2018-11-20 05:30:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:30:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565972  12 58442 19809397 A  ERRORB¢ff'
2018-11-20 05:30:11 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 05:30:31 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 05:30:46 [DEBUG   ] Got command RPT from MCS: ref #565976
2018-11-20 05:30:46 [DEBUG   ] summary = ERROR
2018-11-20 05:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565976  13 58442 19846609 A  ERRORERROR'
2018-11-20 05:30:46 [DEBUG   ] Got command RPT from MCS: ref #565977
2018-11-20 05:30:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   565977  98 58442 19846610 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:30:51 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 05:31:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:31:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367606128000000
2018-11-20 05:31:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:31:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:31:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:31:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:31:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367606128000000
2018-11-20 05:31:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:31:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:31:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:31:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:31:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:31:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:31:06 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:31:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 05:31:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:31:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:31:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:31:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:31:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:31:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:31:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:31:06 [DEBUG   ] Finished updating board flags in 1.605 seconds
2018-11-20 05:31:11 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 05:31:31 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 05:31:51 [DEBUG   ] Finished updating ant stats in 1.005 seconds
2018-11-20 05:32:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:32:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367617888000000
2018-11-20 05:32:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:32:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:32:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:32:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:32:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367617888000000
2018-11-20 05:32:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:32:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:32:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:32:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:32:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:32:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:32:06 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:32:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:32:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:32:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:32:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:32:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:32:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:32:06 [DEBUG   ] Finished updating board flags in 1.606 seconds
2018-11-20 05:32:11 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 05:32:31 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 05:32:51 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:33:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:33:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367629648000000
2018-11-20 05:33:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:33:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:33:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:33:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:33:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367629648000000
2018-11-20 05:33:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:33:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 05:33:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:33:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:33:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:33:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:33:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:33:06 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:33:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:33:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:33:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:33:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:33:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:33:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:33:06 [DEBUG   ] Finished updating board flags in 1.589 seconds
2018-11-20 05:33:11 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-20 05:33:31 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 05:33:51 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:34:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:34:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367641408000000
2018-11-20 05:34:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:34:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:34:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:34:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:34:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367641408000000
2018-11-20 05:34:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 05:34:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:34:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:34:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:34:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:34:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:34:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:34:06 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:34:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:34:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:34:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:34:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:34:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:34:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:34:06 [DEBUG   ] Finished updating board flags in 1.420 seconds
2018-11-20 05:34:11 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:34:31 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:34:51 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 05:35:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:35:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367653168000000
2018-11-20 05:35:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:35:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:35:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:35:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:35:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367653168000000
2018-11-20 05:35:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:35:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 05:35:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:35:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:35:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:35:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:35:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:35:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:35:06 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:35:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:35:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:35:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:35:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:35:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:35:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:35:06 [DEBUG   ] Finished updating board flags in 1.427 seconds
2018-11-20 05:35:08 [DEBUG   ] Got command RPT from MCS: ref #566001
2018-11-20 05:35:08 [DEBUG   ] summary = ERROR
2018-11-20 05:35:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566001  13 58442 20108665 A  ERRORERROR'
2018-11-20 05:35:08 [DEBUG   ] Got command RPT from MCS: ref #566002
2018-11-20 05:35:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 05:35:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566002  71 58442 20108869 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 05:35:09 [DEBUG   ] Got command RPT from MCS: ref #566003
2018-11-20 05:35:09 [DEBUG   ] summary = ERROR
2018-11-20 05:35:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566003  13 58442 20109073 A  ERRORERROR'
2018-11-20 05:35:09 [DEBUG   ] Got command RPT from MCS: ref #566004
2018-11-20 05:35:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 05:35:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566004  71 58442 20109277 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 05:35:09 [DEBUG   ] Got command RPT from MCS: ref #566005
2018-11-20 05:35:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:35:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566005  12 58442 20109469 A  ERRORB¢ff'
2018-11-20 05:35:12 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 05:35:31 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 05:35:46 [DEBUG   ] Got command RPT from MCS: ref #566006
2018-11-20 05:35:46 [DEBUG   ] summary = ERROR
2018-11-20 05:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566006  13 58442 20146613 A  ERRORERROR'
2018-11-20 05:35:46 [DEBUG   ] Got command RPT from MCS: ref #566007
2018-11-20 05:35:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 05:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566007  71 58442 20146614 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 05:35:52 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 05:36:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:36:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367664928000000
2018-11-20 05:36:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:36:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:36:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:36:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:36:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367664928000000
2018-11-20 05:36:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 05:36:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:36:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 05:36:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:36:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:36:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:36:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:36:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:36:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:36:06 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:36:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 05:36:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:36:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:36:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:36:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:36:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:36:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:36:06 [DEBUG   ] Finished updating board flags in 1.415 seconds
2018-11-20 05:36:12 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 05:36:32 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 05:36:52 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 05:37:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:37:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367676688000000
2018-11-20 05:37:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:37:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:37:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:37:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:37:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367676688000000
2018-11-20 05:37:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:37:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:37:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:37:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:37:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:37:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:37:06 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:37:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:37:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:37:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:37:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:37:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:37:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:37:06 [DEBUG   ] Finished updating board flags in 1.424 seconds
2018-11-20 05:37:12 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 05:37:32 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 05:37:52 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 05:38:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:38:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367688448000000
2018-11-20 05:38:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:38:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:38:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:38:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:38:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367688448000000
2018-11-20 05:38:05 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:38:05 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:38:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:38:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:38:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:38:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:38:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:38:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:38:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 05:38:06 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:38:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:38:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:38:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:38:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:38:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:38:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:38:06 [DEBUG   ] Finished updating board flags in 1.429 seconds
2018-11-20 05:38:12 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 05:38:32 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 05:38:52 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:39:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:39:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367700208000000
2018-11-20 05:39:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:39:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:39:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:39:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:39:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367700208000000
2018-11-20 05:39:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 05:39:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:39:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:39:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367700404000000
2018-11-20 05:39:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:39:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:39:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:39:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:39:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367700404000000
2018-11-20 05:39:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:39:06 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:39:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 05:39:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:39:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:39:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:39:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:39:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:39:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 05:39:06 [DEBUG   ] Finished updating board flags in 1.422 seconds
2018-11-20 05:39:12 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:39:32 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:39:52 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 05:40:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:40:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367711968000000
2018-11-20 05:40:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:40:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:40:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:40:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:40:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367711968000000
2018-11-20 05:40:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:40:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:40:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367712164000000
2018-11-20 05:40:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:40:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:40:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:40:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:40:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367712164000000
2018-11-20 05:40:06 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:40:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:40:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:40:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:40:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:40:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:40:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 05:40:06 [DEBUG   ] Finished updating board flags in 1.445 seconds
2018-11-20 05:40:08 [DEBUG   ] Got command RPT from MCS: ref #566029
2018-11-20 05:40:08 [DEBUG   ] summary = ERROR
2018-11-20 05:40:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566029  13 58442 20408753 A  ERRORERROR'
2018-11-20 05:40:08 [DEBUG   ] Got command RPT from MCS: ref #566030
2018-11-20 05:40:08 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 05:40:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566030  79 58442 20408957 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 05:40:09 [DEBUG   ] Got command RPT from MCS: ref #566031
2018-11-20 05:40:09 [DEBUG   ] summary = ERROR
2018-11-20 05:40:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566031  13 58442 20409149 A  ERRORERROR'
2018-11-20 05:40:09 [DEBUG   ] Got command RPT from MCS: ref #566033
2018-11-20 05:40:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 05:40:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566033  79 58442 20409353 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 05:40:09 [DEBUG   ] Got command RPT from MCS: ref #566035
2018-11-20 05:40:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:40:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566035  12 58442 20409557 A  ERRORB¡ff'
2018-11-20 05:40:12 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:40:32 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:40:46 [DEBUG   ] Got command RPT from MCS: ref #566043
2018-11-20 05:40:46 [DEBUG   ] summary = ERROR
2018-11-20 05:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566043  13 58442 20446621 A  ERRORERROR'
2018-11-20 05:40:46 [DEBUG   ] Got command RPT from MCS: ref #566044
2018-11-20 05:40:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 05:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566044  79 58442 20446622 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 05:40:52 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 05:41:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:41:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367723728000000
2018-11-20 05:41:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:41:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:41:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:41:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:41:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367723728000000
2018-11-20 05:41:05 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 05:41:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:41:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:41:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367723924000000
2018-11-20 05:41:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:41:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:41:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:41:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:41:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367723924000000
2018-11-20 05:41:06 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:41:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 05:41:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:41:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:41:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:41:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:41:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:41:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:41:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 05:41:06 [DEBUG   ] Finished updating board flags in 1.438 seconds
2018-11-20 05:41:13 [DEBUG   ] Finished updating ant stats in 1.394 seconds
2018-11-20 05:41:33 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 05:41:53 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 05:42:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:42:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367735488000000
2018-11-20 05:42:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:42:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:42:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:42:05 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:42:05 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367735488000000
2018-11-20 05:42:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:42:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:42:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367735684000000
2018-11-20 05:42:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:42:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:42:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:42:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:42:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367735684000000
2018-11-20 05:42:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:42:06 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:42:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:42:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:42:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:42:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:42:06 [WARNING ] (13, 1, 'bot')
2018-11-20 05:42:06 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:42:06 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 05:42:06 [DEBUG   ] Finished updating board flags in 1.440 seconds
2018-11-20 05:42:13 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 05:42:33 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 05:42:53 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:43:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:43:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:43:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:43:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:43:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:43:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-20 05:43:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:43:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:43:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:43:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:43:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:43:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:43:07 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:43:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:43:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:43:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:43:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:43:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:43:07 [DEBUG   ] Finished updating board flags in 2.228 seconds
2018-11-20 05:43:13 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:43:33 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 05:43:53 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:44:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:44:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:44:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:44:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:44:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:44:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 05:44:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:44:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:44:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:44:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:44:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:44:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:44:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:44:07 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:44:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:44:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:44:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:44:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:44:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:44:07 [DEBUG   ] Finished updating board flags in 2.144 seconds
2018-11-20 05:44:13 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:44:33 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:44:53 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 05:45:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:45:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:45:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:45:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:45:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:45:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:45:06 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 05:45:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:45:06 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 05:45:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:45:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:45:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:45:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:45:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:45:07 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:45:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 05:45:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:45:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:45:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:45:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:45:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:45:07 [DEBUG   ] Finished updating board flags in 2.012 seconds
2018-11-20 05:45:08 [DEBUG   ] Got command RPT from MCS: ref #566064
2018-11-20 05:45:08 [DEBUG   ] summary = ERROR
2018-11-20 05:45:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566064  13 58442 20708825 A  ERRORERROR'
2018-11-20 05:45:09 [DEBUG   ] Got command RPT from MCS: ref #566065
2018-11-20 05:45:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:45:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566065  98 58442 20709029 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:45:09 [DEBUG   ] Got command RPT from MCS: ref #566066
2018-11-20 05:45:09 [DEBUG   ] summary = ERROR
2018-11-20 05:45:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566066  13 58442 20709233 A  ERRORERROR'
2018-11-20 05:45:09 [DEBUG   ] Got command RPT from MCS: ref #566067
2018-11-20 05:45:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:45:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566067  98 58442 20709437 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:45:09 [DEBUG   ] Got command RPT from MCS: ref #566068
2018-11-20 05:45:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:45:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566068  12 58442 20709629 A  ERRORB¢ff'
2018-11-20 05:45:14 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 05:45:34 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 05:45:46 [DEBUG   ] Got command RPT from MCS: ref #566069
2018-11-20 05:45:46 [DEBUG   ] summary = ERROR
2018-11-20 05:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566069  13 58442 20746613 A  ERRORERROR'
2018-11-20 05:45:46 [DEBUG   ] Got command RPT from MCS: ref #566070
2018-11-20 05:45:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566070  98 58442 20746614 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:45:54 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 05:46:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:46:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:46:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:46:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:46:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:46:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:46:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:46:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 05:46:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:46:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:46:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:46:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:46:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:46:07 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:46:07 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 05:46:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:46:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:46:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:46:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:46:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:46:07 [DEBUG   ] Finished updating board flags in 2.005 seconds
2018-11-20 05:46:14 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 05:46:34 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 05:46:54 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 05:47:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:47:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:47:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:47:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:47:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:47:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:47:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:47:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:47:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:47:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:47:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:47:07 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:47:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa7, chip 5, bottom port
2018-11-20 05:47:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:47:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:47:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:47:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:47:07 [WARNING ] (25, 5, 'bot')
2018-11-20 05:47:07 [WARNING ] (25, 5, 'bot')
2018-11-20 05:47:07 [WARNING ] (25, 5, 'bot')
2018-11-20 05:47:07 [WARNING ] (25, 5, 'bot')
2018-11-20 05:47:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:47:07 [DEBUG   ] Finished updating board flags in 1.992 seconds
2018-11-20 05:47:14 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 05:47:34 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 05:47:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 05:48:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:48:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:48:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:48:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:48:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:48:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:48:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:48:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:48:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:48:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:48:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:48:07 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:48:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 05:48:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:48:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:48:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:48:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:48:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:48:07 [DEBUG   ] Finished updating board flags in 1.822 seconds
2018-11-20 05:48:14 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:48:34 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 05:48:54 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 05:49:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:49:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:49:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:49:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:49:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:49:06 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:49:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 05:49:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:49:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 05:49:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:49:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:49:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:49:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:49:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:49:07 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:49:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:49:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:49:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:49:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:49:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:49:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:49:07 [DEBUG   ] Finished updating board flags in 1.784 seconds
2018-11-20 05:49:14 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:49:34 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 05:49:54 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 05:50:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:50:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367829764000000
2018-11-20 05:50:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:50:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:50:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:50:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:50:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367829764000000
2018-11-20 05:50:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:50:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:50:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:50:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:50:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:50:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:50:07 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:50:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:50:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:50:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:50:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:50:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:50:07 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:50:07 [DEBUG   ] Finished updating board flags in 1.626 seconds
2018-11-20 05:50:08 [DEBUG   ] Got command RPT from MCS: ref #566096
2018-11-20 05:50:08 [DEBUG   ] summary = ERROR
2018-11-20 05:50:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566096  13 58442 21008913 A  ERRORERROR'
2018-11-20 05:50:09 [DEBUG   ] Got command RPT from MCS: ref #566097
2018-11-20 05:50:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 05:50:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566097  71 58442 21009118 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 05:50:09 [DEBUG   ] Got command RPT from MCS: ref #566098
2018-11-20 05:50:09 [DEBUG   ] summary = ERROR
2018-11-20 05:50:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566098  13 58442 21009309 A  ERRORERROR'
2018-11-20 05:50:09 [DEBUG   ] Got command RPT from MCS: ref #566100
2018-11-20 05:50:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 05:50:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566100  71 58442 21009513 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 05:50:09 [DEBUG   ] Got command RPT from MCS: ref #566101
2018-11-20 05:50:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:50:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566101  12 58442 21009718 A  ERRORB¢ff'
2018-11-20 05:50:14 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 05:50:35 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 05:50:46 [DEBUG   ] Got command RPT from MCS: ref #566106
2018-11-20 05:50:46 [DEBUG   ] summary = ERROR
2018-11-20 05:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566106  13 58442 21046621 A  ERRORERROR'
2018-11-20 05:50:46 [DEBUG   ] Got command RPT from MCS: ref #566107
2018-11-20 05:50:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 05:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566107  71 58442 21046622 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 05:50:55 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 05:51:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:51:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367841524000000
2018-11-20 05:51:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:51:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:51:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:51:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:51:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367841524000000
2018-11-20 05:51:06 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:51:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:51:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:51:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:51:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:51:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:51:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:51:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:51:07 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:51:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 05:51:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:51:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:51:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:51:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:51:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:51:07 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:51:07 [DEBUG   ] Finished updating board flags in 1.597 seconds
2018-11-20 05:51:15 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 05:51:35 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 05:51:55 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 05:52:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:52:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367853284000000
2018-11-20 05:52:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:52:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:52:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:52:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:52:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367853284000000
2018-11-20 05:52:06 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:52:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:52:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:52:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:52:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:52:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:52:07 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 05:52:07 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:52:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:52:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:52:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:52:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:52:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:52:07 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 05:52:07 [DEBUG   ] Finished updating board flags in 1.383 seconds
2018-11-20 05:52:15 [DEBUG   ] Finished updating ant stats in 0.997 seconds
2018-11-20 05:52:35 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:52:55 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-20 05:53:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:53:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367865044000000
2018-11-20 05:53:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:53:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:53:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:53:06 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:53:06 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367865044000000
2018-11-20 05:53:06 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 05:53:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:53:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:53:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367865240000000
2018-11-20 05:53:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:53:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:53:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:53:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:53:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367865240000000
2018-11-20 05:53:07 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:53:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:53:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:53:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:53:07 [WARNING ] (13, 1, 'bot')
2018-11-20 05:53:07 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:53:07 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 05:53:07 [DEBUG   ] Finished updating board flags in 1.379 seconds
2018-11-20 05:53:15 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 05:53:35 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 05:53:55 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:54:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:54:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:54:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:54:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:54:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:54:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:54:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:54:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:54:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:54:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:54:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:54:08 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:54:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 05:54:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:54:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:54:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:54:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:54:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:54:08 [DEBUG   ] Finished updating board flags in 2.033 seconds
2018-11-20 05:54:15 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 05:54:35 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 05:54:56 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 05:55:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:55:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:55:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:55:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:55:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:55:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 05:55:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 05:55:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:55:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:55:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:55:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:55:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:55:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:55:08 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:55:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:55:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:55:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:55:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:55:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:55:08 [DEBUG   ] Finished updating board flags in 1.988 seconds
2018-11-20 05:55:08 [DEBUG   ] Got command RPT from MCS: ref #566127
2018-11-20 05:55:08 [DEBUG   ] summary = ERROR
2018-11-20 05:55:08 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566127  13 58442 21308993 A  ERRORERROR'
2018-11-20 05:55:09 [DEBUG   ] Got command RPT from MCS: ref #566128
2018-11-20 05:55:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:55:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566128  98 58442 21309197 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:55:09 [DEBUG   ] Got command RPT from MCS: ref #566129
2018-11-20 05:55:09 [DEBUG   ] summary = ERROR
2018-11-20 05:55:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566129  13 58442 21309389 A  ERRORERROR'
2018-11-20 05:55:09 [DEBUG   ] Got command RPT from MCS: ref #566130
2018-11-20 05:55:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:55:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566130  98 58442 21309593 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:55:09 [DEBUG   ] Got command RPT from MCS: ref #566131
2018-11-20 05:55:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 05:55:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566131  12 58442 21309797 A  ERRORB¡ff'
2018-11-20 05:55:16 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 05:55:36 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 05:55:46 [DEBUG   ] Got command RPT from MCS: ref #566136
2018-11-20 05:55:46 [DEBUG   ] summary = ERROR
2018-11-20 05:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566136  13 58442 21346617 A  ERRORERROR'
2018-11-20 05:55:46 [DEBUG   ] Got command RPT from MCS: ref #566137
2018-11-20 05:55:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 05:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566137  98 58442 21346618 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 05:55:56 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 05:56:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:56:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:56:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:56:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:56:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:56:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 05:56:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:56:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:56:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:56:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:56:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:56:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:56:08 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 05:56:08 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 05:56:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:56:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:56:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:56:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:56:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:56:08 [DEBUG   ] Finished updating board flags in 1.822 seconds
2018-11-20 05:56:16 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 05:56:36 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 05:56:56 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 05:57:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:57:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:57:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:57:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:57:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:57:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:57:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:57:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:57:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:57:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:57:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:57:08 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:57:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:57:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:57:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:57:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:57:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:57:08 [DEBUG   ] Finished updating board flags in 1.805 seconds
2018-11-20 05:57:16 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:57:36 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 05:57:56 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 05:58:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:58:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:58:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:58:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:58:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:58:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 05:58:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:58:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:58:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:58:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:58:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:58:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:58:08 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:58:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:58:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:58:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:58:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:58:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:58:08 [DEBUG   ] Finished updating board flags in 1.814 seconds
2018-11-20 05:58:16 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 05:58:36 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 05:58:56 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 05:59:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 05:59:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 05:59:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 05:59:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 05:59:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 05:59:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 05:59:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 05:59:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 05:59:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 05:59:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 05:59:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 05:59:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 05:59:08 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 05:59:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:59:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:59:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:59:08 [WARNING ] (13, 1, 'bot')
2018-11-20 05:59:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 05:59:08 [DEBUG   ] Finished updating board flags in 1.768 seconds
2018-11-20 05:59:16 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 05:59:36 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 05:59:57 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 06:00:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:00:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367947560000000
2018-11-20 06:00:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:00:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:00:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:00:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:00:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367947560000000
2018-11-20 06:00:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:00:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:00:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:00:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:00:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:00:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:00:08 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:00:08 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:00:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:00:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:00:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:00:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:00:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:00:08 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:00:08 [DEBUG   ] Finished updating board flags in 1.630 seconds
2018-11-20 06:00:09 [DEBUG   ] Got command RPT from MCS: ref #566159
2018-11-20 06:00:09 [DEBUG   ] summary = ERROR
2018-11-20 06:00:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566159  13 58442 21609077 A  ERRORERROR'
2018-11-20 06:00:09 [DEBUG   ] Got command RPT from MCS: ref #566160
2018-11-20 06:00:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:00:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566160  71 58442 21609269 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:00:09 [DEBUG   ] Got command RPT from MCS: ref #566161
2018-11-20 06:00:09 [DEBUG   ] summary = ERROR
2018-11-20 06:00:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566161  13 58442 21609473 A  ERRORERROR'
2018-11-20 06:00:09 [DEBUG   ] Got command RPT from MCS: ref #566163
2018-11-20 06:00:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:00:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566163  71 58442 21609677 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:00:09 [DEBUG   ] Got command RPT from MCS: ref #566165
2018-11-20 06:00:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:00:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566165  12 58442 21609881 A  ERRORB¡ff'
2018-11-20 06:00:17 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 06:00:37 [DEBUG   ] Finished updating ant stats in 1.051 seconds
2018-11-20 06:00:46 [DEBUG   ] Got command RPT from MCS: ref #566169
2018-11-20 06:00:46 [DEBUG   ] summary = ERROR
2018-11-20 06:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566169  13 58442 21646617 A  ERRORERROR'
2018-11-20 06:00:46 [DEBUG   ] Got command RPT from MCS: ref #566170
2018-11-20 06:00:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566170  71 58442 21646618 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:00:57 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 06:01:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:01:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367959320000000
2018-11-20 06:01:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:01:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:01:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:01:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:01:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367959320000000
2018-11-20 06:01:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 06:01:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:01:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:01:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:01:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:01:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:01:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:01:08 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 06:01:08 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:01:08 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:01:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:01:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:01:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:01:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:01:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:01:08 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:01:08 [DEBUG   ] Finished updating board flags in 1.593 seconds
2018-11-20 06:01:17 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 06:01:37 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 06:01:57 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 06:02:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:02:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367971080000000
2018-11-20 06:02:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:02:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:02:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:02:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:02:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367971080000000
2018-11-20 06:02:07 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 06:02:07 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:02:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:02:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:02:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:02:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:02:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:02:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-20 06:02:08 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:02:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:02:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:02:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:02:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:02:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:02:08 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:02:08 [DEBUG   ] Finished updating board flags in 1.394 seconds
2018-11-20 06:02:17 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 06:02:37 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 06:02:57 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 06:03:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:03:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302367982840000000
2018-11-20 06:03:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:03:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:03:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:03:07 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:03:07 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302367982840000000
2018-11-20 06:03:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:03:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:03:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302367983036000000
2018-11-20 06:03:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:03:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:03:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:03:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:03:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302367983036000000
2018-11-20 06:03:08 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:03:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 06:03:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:03:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:03:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:03:08 [WARNING ] (13, 1, 'bot')
2018-11-20 06:03:08 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:03:08 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 06:03:08 [DEBUG   ] Finished updating board flags in 1.372 seconds
2018-11-20 06:03:17 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 06:03:37 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 06:03:57 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 06:04:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:04:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:04:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:04:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:04:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:04:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:04:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:04:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:04:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:04:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:04:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:04:09 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:04:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:04:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:04:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:04:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:04:09 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:04:09 [DEBUG   ] Finished updating board flags in 2.040 seconds
2018-11-20 06:04:17 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 06:04:37 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 06:04:58 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 06:05:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:05:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:05:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:05:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:05:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:05:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 06:05:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:05:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:05:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:05:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:05:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:05:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:05:09 [DEBUG   ] Got command RPT from MCS: ref #566194
2018-11-20 06:05:09 [DEBUG   ] summary = ERROR
2018-11-20 06:05:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566194  13 58442 21909153 A  ERRORERROR'
2018-11-20 06:05:09 [DEBUG   ] Got command RPT from MCS: ref #566195
2018-11-20 06:05:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:05:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566195  98 58442 21909362 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:05:09 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:05:09 [DEBUG   ] Got command RPT from MCS: ref #566196
2018-11-20 06:05:09 [DEBUG   ] summary = ERROR
2018-11-20 06:05:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566196  13 58442 21909558 A  ERRORERROR'
2018-11-20 06:05:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:05:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:05:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:05:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:05:09 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:05:09 [DEBUG   ] Finished updating board flags in 1.985 seconds
2018-11-20 06:05:09 [DEBUG   ] Got command RPT from MCS: ref #566197
2018-11-20 06:05:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:05:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566197  98 58442 21909749 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:05:09 [DEBUG   ] Got command RPT from MCS: ref #566198
2018-11-20 06:05:09 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:05:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566198  12 58442 21909953 A  ERRORB¢ff'
2018-11-20 06:05:17 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 06:05:37 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-20 06:05:46 [DEBUG   ] Got command RPT from MCS: ref #566199
2018-11-20 06:05:46 [DEBUG   ] summary = ERROR
2018-11-20 06:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566199  13 58442 21946625 A  ERRORERROR'
2018-11-20 06:05:46 [DEBUG   ] Got command RPT from MCS: ref #566200
2018-11-20 06:05:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566200  98 58442 21946626 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:05:58 [DEBUG   ] Finished updating ant stats in 1.395 seconds
2018-11-20 06:06:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:06:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:06:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:06:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:06:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:06:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:06:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 06:06:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:06:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:06:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:06:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:06:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:06:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:06:09 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:06:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:06:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:06:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:06:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:06:09 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:06:09 [DEBUG   ] Finished updating board flags in 1.868 seconds
2018-11-20 06:06:18 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 06:06:38 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 06:06:58 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 06:07:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:07:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:07:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:07:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:07:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:07:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:07:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 06:07:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 06:07:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:07:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:07:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:07:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:07:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:07:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:07:09 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:07:09 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:07:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:07:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:07:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:07:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:07:09 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:07:09 [DEBUG   ] Finished updating board flags in 1.737 seconds
2018-11-20 06:07:18 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 06:07:38 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 06:07:58 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 06:08:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:08:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368041836000000
2018-11-20 06:08:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:08:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:08:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:08:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:08:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368041836000000
2018-11-20 06:08:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:08:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:08:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:08:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:08:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:08:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:08:09 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:08:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:08:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:08:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:08:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:08:09 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:08:09 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:08:09 [DEBUG   ] Finished updating board flags in 1.620 seconds
2018-11-20 06:08:18 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 06:08:38 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 06:08:58 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 06:09:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:09:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368053596000000
2018-11-20 06:09:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:09:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:09:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:09:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:09:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368053596000000
2018-11-20 06:09:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:09:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:09:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:09:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:09:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:09:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:09:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 06:09:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 06:09:09 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:09:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:09:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:09:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:09:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:09:09 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:09:09 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:09:09 [DEBUG   ] Finished updating board flags in 1.587 seconds
2018-11-20 06:09:18 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 06:09:38 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 06:09:58 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 06:10:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:10:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368065356000000
2018-11-20 06:10:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:10:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:10:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:10:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:10:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368065356000000
2018-11-20 06:10:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:10:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:10:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:10:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:10:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:10:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:10:09 [DEBUG   ] Got command RPT from MCS: ref #566222
2018-11-20 06:10:09 [DEBUG   ] summary = ERROR
2018-11-20 06:10:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566222  13 58442 22209229 A  ERRORERROR'
2018-11-20 06:10:09 [DEBUG   ] Got command RPT from MCS: ref #566223
2018-11-20 06:10:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:10:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566223  71 58442 22209433 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:10:09 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:10:09 [DEBUG   ] Got command RPT from MCS: ref #566224
2018-11-20 06:10:09 [DEBUG   ] summary = ERROR
2018-11-20 06:10:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566224  13 58442 22209634 A  ERRORERROR'
2018-11-20 06:10:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:10:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:10:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:10:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:10:09 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:10:09 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:10:09 [DEBUG   ] Finished updating board flags in 1.459 seconds
2018-11-20 06:10:09 [DEBUG   ] Got command RPT from MCS: ref #566226
2018-11-20 06:10:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:10:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566226  71 58442 22209837 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:10:10 [DEBUG   ] Got command RPT from MCS: ref #566228
2018-11-20 06:10:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:10:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566228  12 58442 22210029 A  ERRORB¢ff'
2018-11-20 06:10:18 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 06:10:38 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 06:10:46 [DEBUG   ] Got command RPT from MCS: ref #566236
2018-11-20 06:10:46 [DEBUG   ] summary = ERROR
2018-11-20 06:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566236  13 58442 22246617 A  ERRORERROR'
2018-11-20 06:10:46 [DEBUG   ] Got command RPT from MCS: ref #566237
2018-11-20 06:10:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566237  71 58442 22246618 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:10:58 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 06:11:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:11:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368077116000000
2018-11-20 06:11:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:11:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:11:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:11:08 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:11:08 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368077116000000
2018-11-20 06:11:08 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 06:11:08 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:11:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:11:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:11:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:11:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:11:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:11:09 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:11:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:11:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:11:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:11:09 [WARNING ] (13, 1, 'bot')
2018-11-20 06:11:09 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:11:09 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:11:09 [DEBUG   ] Finished updating board flags in 1.367 seconds
2018-11-20 06:11:19 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 06:11:38 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 06:11:59 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 06:12:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:12:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:12:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:12:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:12:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:12:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 06:12:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:12:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:12:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:12:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:12:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:12:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:12:10 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:12:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 06:12:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:12:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:12:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:12:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:12:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:12:10 [DEBUG   ] Finished updating board flags in 2.206 seconds
2018-11-20 06:12:19 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 06:12:39 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 06:12:59 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 06:13:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:13:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:13:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:13:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:13:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:13:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 06:13:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:13:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:13:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:13:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:13:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:13:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:13:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 06:13:10 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:13:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:13:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:13:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:13:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:13:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:13:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:13:10 [DEBUG   ] Finished updating board flags in 2.208 seconds
2018-11-20 06:13:19 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 06:13:39 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 06:13:59 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 06:14:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:14:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:14:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:14:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:14:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:14:09 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:14:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 06:14:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:14:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:14:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:14:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:14:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:14:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:14:10 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:14:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:14:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:14:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:14:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:14:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:14:10 [DEBUG   ] Finished updating board flags in 2.166 seconds
2018-11-20 06:14:19 [DEBUG   ] Finished updating ant stats in 0.995 seconds
2018-11-20 06:14:39 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 06:14:59 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 06:15:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:15:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:15:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:15:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:15:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:15:09 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:15:09 [DEBUG   ] Got command RPT from MCS: ref #566257
2018-11-20 06:15:09 [DEBUG   ] summary = ERROR
2018-11-20 06:15:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566257  13 58442 22509314 A  ERRORERROR'
2018-11-20 06:15:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 06:15:09 [DEBUG   ] Got command RPT from MCS: ref #566258
2018-11-20 06:15:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:15:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566258  98 58442 22509518 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:15:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:15:09 [DEBUG   ] Got command RPT from MCS: ref #566259
2018-11-20 06:15:09 [DEBUG   ] summary = ERROR
2018-11-20 06:15:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566259  13 58442 22509721 A  ERRORERROR'
2018-11-20 06:15:09 [DEBUG   ] Got command RPT from MCS: ref #566260
2018-11-20 06:15:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:15:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566260  98 58442 22509913 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:15:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:15:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:15:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:15:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:15:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:15:10 [DEBUG   ] Got command RPT from MCS: ref #566261
2018-11-20 06:15:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:15:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566261  12 58442 22510118 A  ERRORBš™š'
2018-11-20 06:15:10 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:15:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:15:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:15:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:15:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:15:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:15:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:15:10 [DEBUG   ] Finished updating board flags in 2.012 seconds
2018-11-20 06:15:19 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 06:15:39 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 06:15:46 [DEBUG   ] Got command RPT from MCS: ref #566262
2018-11-20 06:15:46 [DEBUG   ] summary = ERROR
2018-11-20 06:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566262  13 58442 22546621 A  ERRORERROR'
2018-11-20 06:15:46 [DEBUG   ] Got command RPT from MCS: ref #566263
2018-11-20 06:15:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566263  98 58442 22546622 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:15:59 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 06:16:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:16:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:16:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:16:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:16:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:16:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:16:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:16:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:16:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:16:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:16:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:16:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:16:10 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:16:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:16:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:16:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:16:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:16:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:16:10 [DEBUG   ] Finished updating board flags in 2.009 seconds
2018-11-20 06:16:19 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 06:16:39 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 06:16:59 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 06:17:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:17:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:17:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:17:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:17:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:17:09 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 06:17:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:17:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:17:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:17:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:17:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:17:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:17:10 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:17:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:17:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:17:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:17:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:17:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:17:10 [DEBUG   ] Finished updating board flags in 1.989 seconds
2018-11-20 06:17:20 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 06:17:40 [DEBUG   ] Finished updating ant stats in 1.404 seconds
2018-11-20 06:18:00 [DEBUG   ] Finished updating ant stats in 1.395 seconds
2018-11-20 06:18:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:18:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:18:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:18:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:18:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:18:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:18:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:18:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:18:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:18:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:18:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:18:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:18:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:18:10 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:18:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:18:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:18:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:18:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:18:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:18:10 [DEBUG   ] Finished updating board flags in 1.831 seconds
2018-11-20 06:18:20 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 06:18:40 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 06:19:00 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 06:19:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:19:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:19:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:19:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:19:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:19:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:19:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:19:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:19:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:19:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:19:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:19:10 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:19:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:19:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:19:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:19:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:19:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:19:10 [DEBUG   ] Finished updating board flags in 1.762 seconds
2018-11-20 06:19:20 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 06:19:40 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 06:20:00 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 06:20:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:20:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368183152000000
2018-11-20 06:20:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:20:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:20:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:20:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:20:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368183152000000
2018-11-20 06:20:09 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:20:09 [DEBUG   ] Got command RPT from MCS: ref #566289
2018-11-20 06:20:09 [DEBUG   ] summary = ERROR
2018-11-20 06:20:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566289  13 58442 22809401 A  ERRORERROR'
2018-11-20 06:20:09 [DEBUG   ] Got command RPT from MCS: ref #566290
2018-11-20 06:20:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:20:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566290  98 58442 22809593 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:20:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:20:09 [DEBUG   ] Got command RPT from MCS: ref #566291
2018-11-20 06:20:09 [DEBUG   ] summary = ERROR
2018-11-20 06:20:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566291  13 58442 22809797 A  ERRORERROR'
2018-11-20 06:20:10 [DEBUG   ] Got command RPT from MCS: ref #566293
2018-11-20 06:20:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:20:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566293  98 58442 22810001 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:20:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:20:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:20:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:20:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:20:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:20:10 [DEBUG   ] Got command RPT from MCS: ref #566294
2018-11-20 06:20:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:20:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566294  12 58442 22810205 A  ERRORB™™š'
2018-11-20 06:20:10 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:20:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:20:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:20:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:20:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:20:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:20:10 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:20:10 [DEBUG   ] Finished updating board flags in 1.625 seconds
2018-11-20 06:20:20 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 06:20:40 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 06:20:46 [DEBUG   ] Got command RPT from MCS: ref #566299
2018-11-20 06:20:46 [DEBUG   ] summary = ERROR
2018-11-20 06:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566299  13 58442 22846629 A  ERRORERROR'
2018-11-20 06:20:46 [DEBUG   ] Got command RPT from MCS: ref #566300
2018-11-20 06:20:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566300  71 58442 22846630 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:21:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 06:21:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:21:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368194912000000
2018-11-20 06:21:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:21:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:21:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:21:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:21:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368194912000000
2018-11-20 06:21:09 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:21:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:21:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:21:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:21:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:21:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:21:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:21:10 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:21:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:21:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:21:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:21:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:21:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:21:10 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:21:10 [DEBUG   ] Finished updating board flags in 1.575 seconds
2018-11-20 06:21:20 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 06:21:40 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 06:22:00 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 06:22:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:22:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368206672000000
2018-11-20 06:22:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:22:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:22:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:22:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:22:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368206672000000
2018-11-20 06:22:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:22:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 06:22:09 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 06:22:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:22:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:22:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:22:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:22:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:22:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:22:10 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:22:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:22:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:22:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:22:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:22:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:22:10 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:22:10 [DEBUG   ] Finished updating board flags in 1.416 seconds
2018-11-20 06:22:20 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 06:22:40 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 06:23:00 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 06:23:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:23:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368218432000000
2018-11-20 06:23:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:23:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:23:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:23:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:23:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368218432000000
2018-11-20 06:23:09 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 06:23:09 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:23:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:23:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:23:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:23:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:23:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:23:10 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:23:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:23:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:23:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:23:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:23:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:23:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:23:10 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:23:10 [DEBUG   ] Finished updating board flags in 1.392 seconds
2018-11-20 06:23:20 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 06:23:40 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 06:24:01 [DEBUG   ] Finished updating ant stats in 1.402 seconds
2018-11-20 06:24:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:24:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368230192000000
2018-11-20 06:24:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:24:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:24:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:24:09 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:24:09 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368230192000000
2018-11-20 06:24:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:24:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:24:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302368230388000000
2018-11-20 06:24:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:24:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:24:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:24:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:24:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302368230388000000
2018-11-20 06:24:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 06:24:10 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:24:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:24:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:24:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:24:10 [WARNING ] (13, 1, 'bot')
2018-11-20 06:24:10 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:24:10 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 06:24:10 [DEBUG   ] Finished updating board flags in 1.399 seconds
2018-11-20 06:24:21 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 06:24:41 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 06:25:01 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 06:25:09 [DEBUG   ] Got command RPT from MCS: ref #566320
2018-11-20 06:25:09 [DEBUG   ] summary = ERROR
2018-11-20 06:25:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566320  13 58442 23109473 A  ERRORERROR'
2018-11-20 06:25:09 [DEBUG   ] Got command RPT from MCS: ref #566321
2018-11-20 06:25:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 06:25:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566321  79 58442 23109677 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 06:25:09 [DEBUG   ] Got command RPT from MCS: ref #566322
2018-11-20 06:25:09 [DEBUG   ] summary = ERROR
2018-11-20 06:25:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566322  13 58442 23109881 A  ERRORERROR'
2018-11-20 06:25:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:25:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:25:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:25:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:25:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:25:10 [DEBUG   ] Got command RPT from MCS: ref #566323
2018-11-20 06:25:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 06:25:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566323  79 58442 23110077 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 06:25:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 06:25:10 [DEBUG   ] Got command RPT from MCS: ref #566324
2018-11-20 06:25:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:25:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566324  12 58442 23110282 A  ERRORB�ÌÍ'
2018-11-20 06:25:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:25:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:25:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:25:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:25:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:25:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:25:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 06:25:11 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:25:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:25:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:25:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:25:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:25:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:25:11 [DEBUG   ] Finished updating board flags in 2.018 seconds
2018-11-20 06:25:21 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 06:25:41 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 06:25:46 [DEBUG   ] Got command RPT from MCS: ref #566329
2018-11-20 06:25:46 [DEBUG   ] summary = ERROR
2018-11-20 06:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566329  13 58442 23146629 A  ERRORERROR'
2018-11-20 06:25:46 [DEBUG   ] Got command RPT from MCS: ref #566330
2018-11-20 06:25:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566330  98 58442 23146630 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:26:01 [DEBUG   ] Finished updating ant stats in 1.005 seconds
2018-11-20 06:26:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:26:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:26:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:26:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:26:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:26:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:26:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:26:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:26:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:26:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:26:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:26:11 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:26:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:26:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:26:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:26:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:26:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:26:11 [DEBUG   ] Finished updating board flags in 2.009 seconds
2018-11-20 06:26:21 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 06:26:41 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-20 06:27:01 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 06:27:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:27:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:27:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:27:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:27:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:27:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:27:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:27:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:27:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:27:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:27:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:27:11 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:27:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:27:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:27:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:27:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:27:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:27:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:27:11 [DEBUG   ] Finished updating board flags in 1.983 seconds
2018-11-20 06:27:21 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 06:27:41 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 06:28:01 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 06:28:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:28:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:28:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:28:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:28:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:28:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:28:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:28:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:28:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:28:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:28:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:28:11 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:28:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:28:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:28:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:28:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:28:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:28:11 [DEBUG   ] Finished updating board flags in 1.813 seconds
2018-11-20 06:28:21 [DEBUG   ] Finished updating ant stats in 0.595 seconds
2018-11-20 06:28:41 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 06:29:02 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 06:29:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:29:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:29:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:29:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:29:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:29:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:29:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 06:29:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 06:29:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:29:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:29:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:29:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:29:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:29:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:29:11 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:29:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:29:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:29:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:29:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:29:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:29:11 [DEBUG   ] Finished updating board flags in 1.802 seconds
2018-11-20 06:29:22 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 06:29:42 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 06:30:02 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 06:30:09 [DEBUG   ] Got command RPT from MCS: ref #566352
2018-11-20 06:30:09 [DEBUG   ] summary = ERROR
2018-11-20 06:30:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566352  13 58442 23409549 A  ERRORERROR'
2018-11-20 06:30:09 [DEBUG   ] Got command RPT from MCS: ref #566353
2018-11-20 06:30:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:30:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566353  98 58442 23409753 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:30:09 [DEBUG   ] Got command RPT from MCS: ref #566354
2018-11-20 06:30:09 [DEBUG   ] summary = ERROR
2018-11-20 06:30:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566354  13 58442 23409957 A  ERRORERROR'
2018-11-20 06:30:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:30:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:30:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:30:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:30:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:30:10 [DEBUG   ] Got command RPT from MCS: ref #566355
2018-11-20 06:30:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:30:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566355  98 58442 23410161 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:30:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 06:30:10 [DEBUG   ] Got command RPT from MCS: ref #566357
2018-11-20 06:30:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:30:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566357  12 58442 23410361 A  ERRORB‹ÌÍ'
2018-11-20 06:30:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:30:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:30:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:30:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:30:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:30:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:30:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:30:11 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:30:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:30:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:30:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:30:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:30:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:30:11 [DEBUG   ] Finished updating board flags in 1.795 seconds
2018-11-20 06:30:22 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 06:30:42 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 06:30:46 [DEBUG   ] Got command RPT from MCS: ref #566362
2018-11-20 06:30:46 [DEBUG   ] summary = ERROR
2018-11-20 06:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566362  13 58442 23446629 A  ERRORERROR'
2018-11-20 06:30:46 [DEBUG   ] Got command RPT from MCS: ref #566363
2018-11-20 06:30:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566363  98 58442 23446630 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:31:02 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 06:31:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:31:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368312708000000
2018-11-20 06:31:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:31:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:31:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:31:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:31:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368312708000000
2018-11-20 06:31:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:31:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:31:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:31:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:31:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:31:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:31:11 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:31:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:31:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:31:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:31:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:31:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:31:11 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:31:11 [DEBUG   ] Finished updating board flags in 1.611 seconds
2018-11-20 06:31:22 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 06:31:42 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 06:32:02 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 06:32:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:32:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368324468000000
2018-11-20 06:32:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:32:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:32:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:32:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:32:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368324468000000
2018-11-20 06:32:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 06:32:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:32:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:32:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:32:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:32:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:32:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:32:11 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 06:32:11 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:32:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:32:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:32:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:32:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:32:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:32:11 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:32:11 [DEBUG   ] Finished updating board flags in 1.630 seconds
2018-11-20 06:32:22 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 06:32:42 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 06:33:02 [DEBUG   ] Finished updating ant stats in 0.588 seconds
2018-11-20 06:33:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:33:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368336228000000
2018-11-20 06:33:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:33:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:33:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:33:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:33:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368336228000000
2018-11-20 06:33:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:33:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 06:33:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:33:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:33:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:33:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:33:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:33:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:33:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:33:11 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:33:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:33:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:33:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:33:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:33:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:33:11 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:33:11 [DEBUG   ] Finished updating board flags in 1.561 seconds
2018-11-20 06:33:23 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-20 06:33:43 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 06:34:03 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 06:34:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:34:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368347988000000
2018-11-20 06:34:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:34:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:34:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:34:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:34:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368347988000000
2018-11-20 06:34:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:34:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:34:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:34:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:34:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:34:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:34:11 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:34:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:34:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:34:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:34:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:34:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:34:11 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:34:11 [DEBUG   ] Finished updating board flags in 1.410 seconds
2018-11-20 06:34:23 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 06:34:43 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 06:35:03 [DEBUG   ] Finished updating ant stats in 1.372 seconds
2018-11-20 06:35:09 [DEBUG   ] Got command RPT from MCS: ref #566387
2018-11-20 06:35:09 [DEBUG   ] summary = ERROR
2018-11-20 06:35:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566387  13 58442 23709651 A  ERRORERROR'
2018-11-20 06:35:09 [DEBUG   ] Got command RPT from MCS: ref #566388
2018-11-20 06:35:09 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:35:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566388  71 58442 23709833 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:35:10 [DEBUG   ] Got command RPT from MCS: ref #566389
2018-11-20 06:35:10 [DEBUG   ] summary = ERROR
2018-11-20 06:35:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566389  13 58442 23710051 A  ERRORERROR'
2018-11-20 06:35:10 [DEBUG   ] Got command RPT from MCS: ref #566390
2018-11-20 06:35:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:35:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566390  71 58442 23710241 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:35:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:35:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368359748000000
2018-11-20 06:35:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:35:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:35:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:35:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:35:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368359748000000
2018-11-20 06:35:10 [DEBUG   ] Got command RPT from MCS: ref #566391
2018-11-20 06:35:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:35:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566391  12 58442 23710434 A  ERRORB‰ÌÍ'
2018-11-20 06:35:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 06:35:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:35:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:35:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:35:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:35:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:35:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:35:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 06:35:11 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:35:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:35:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:35:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:35:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:35:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:35:11 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:35:11 [DEBUG   ] Finished updating board flags in 1.417 seconds
2018-11-20 06:35:23 [DEBUG   ] Finished updating ant stats in 0.831 seconds
2018-11-20 06:35:43 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 06:35:46 [DEBUG   ] Got command RPT from MCS: ref #566392
2018-11-20 06:35:46 [DEBUG   ] summary = ERROR
2018-11-20 06:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566392  13 58442 23746621 A  ERRORERROR'
2018-11-20 06:35:46 [DEBUG   ] Got command RPT from MCS: ref #566393
2018-11-20 06:35:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566393  71 58442 23746622 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:36:03 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 06:36:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:36:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368371508000000
2018-11-20 06:36:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:36:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:36:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:36:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:36:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368371508000000
2018-11-20 06:36:10 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:36:10 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:36:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:36:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:36:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:36:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:36:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:36:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 06:36:11 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:36:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:36:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:36:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:36:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:36:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:36:11 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:36:11 [DEBUG   ] Finished updating board flags in 1.378 seconds
2018-11-20 06:36:23 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 06:36:43 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 06:37:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 06:37:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:37:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368383268000000
2018-11-20 06:37:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:37:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:37:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:37:10 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:37:10 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368383268000000
2018-11-20 06:37:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:37:10 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 06:37:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:37:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:37:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302368383464000000
2018-11-20 06:37:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:37:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:37:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:37:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:37:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302368383464000000
2018-11-20 06:37:11 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:37:11 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:37:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:37:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:37:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:37:11 [WARNING ] (13, 1, 'bot')
2018-11-20 06:37:11 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:37:11 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 06:37:11 [DEBUG   ] Finished updating board flags in 1.368 seconds
2018-11-20 06:37:23 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 06:37:43 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 06:38:03 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 06:38:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:38:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:38:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:38:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:38:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:38:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:38:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:38:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:38:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:38:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:38:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:38:12 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:38:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:38:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:38:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:38:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:38:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:38:12 [DEBUG   ] Finished updating board flags in 2.061 seconds
2018-11-20 06:38:24 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 06:38:44 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 06:39:04 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 06:39:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:39:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:39:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:39:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:39:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:39:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 06:39:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:39:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:39:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:39:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:39:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:39:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:39:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 06:39:12 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:39:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:39:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:39:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:39:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:39:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:39:12 [DEBUG   ] Finished updating board flags in 1.974 seconds
2018-11-20 06:39:24 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 06:39:44 [DEBUG   ] Finished updating ant stats in 1.005 seconds
2018-11-20 06:40:04 [DEBUG   ] Finished updating ant stats in 1.378 seconds
2018-11-20 06:40:09 [DEBUG   ] Got command RPT from MCS: ref #566415
2018-11-20 06:40:09 [DEBUG   ] summary = ERROR
2018-11-20 06:40:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566415  13 58442 24009717 A  ERRORERROR'
2018-11-20 06:40:09 [DEBUG   ] Got command RPT from MCS: ref #566416
2018-11-20 06:40:09 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:40:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566416  98 58442 24009921 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:40:10 [DEBUG   ] Got command RPT from MCS: ref #566417
2018-11-20 06:40:10 [DEBUG   ] summary = ERROR
2018-11-20 06:40:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566417  13 58442 24010117 A  ERRORERROR'
2018-11-20 06:40:10 [DEBUG   ] Got command RPT from MCS: ref #566419
2018-11-20 06:40:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:40:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566419  98 58442 24010313 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:40:10 [DEBUG   ] Got command RPT from MCS: ref #566421
2018-11-20 06:40:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:40:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566421  12 58442 24010521 A  ERRORB ff'
2018-11-20 06:40:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:40:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:40:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:40:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:40:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:40:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:40:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:40:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:40:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:40:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:40:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:40:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:40:12 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:40:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:40:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:40:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:40:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:40:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:40:12 [DEBUG   ] Finished updating board flags in 1.814 seconds
2018-11-20 06:40:24 [DEBUG   ] Finished updating ant stats in 0.819 seconds
2018-11-20 06:40:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 06:40:46 [DEBUG   ] Got command RPT from MCS: ref #566429
2018-11-20 06:40:46 [DEBUG   ] summary = ERROR
2018-11-20 06:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566429  13 58442 24046628 A  ERRORERROR'
2018-11-20 06:40:46 [DEBUG   ] Got command RPT from MCS: ref #566430
2018-11-20 06:40:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566430  98 58442 24046628 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:41:04 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 06:41:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:41:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:41:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:41:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:41:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:41:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:41:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:41:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:41:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:41:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:41:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:41:12 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:41:12 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:41:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:41:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:41:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:41:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:41:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:41:12 [DEBUG   ] Finished updating board flags in 1.810 seconds
2018-11-20 06:41:24 [DEBUG   ] Finished updating ant stats in 0.797 seconds
2018-11-20 06:41:44 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 06:42:04 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 06:42:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:42:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:42:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:42:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:42:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:42:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 06:42:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:42:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:42:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:42:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:42:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:42:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:42:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 06:42:12 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:42:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:42:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:42:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:42:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:42:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:42:12 [DEBUG   ] Finished updating board flags in 1.787 seconds
2018-11-20 06:42:25 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 06:42:44 [DEBUG   ] Finished updating ant stats in 0.408 seconds
2018-11-20 06:43:05 [DEBUG   ] Finished updating ant stats in 1.395 seconds
2018-11-20 06:43:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:43:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368454024000000
2018-11-20 06:43:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:43:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:43:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:43:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:43:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368454024000000
2018-11-20 06:43:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 06:43:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 06:43:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:43:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:43:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:43:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:43:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:43:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:43:12 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:43:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:43:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 06:43:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:43:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:43:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:43:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:43:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:43:12 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:43:12 [DEBUG   ] Finished updating board flags in 1.618 seconds
2018-11-20 06:43:25 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 06:43:45 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 06:44:05 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 06:44:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:44:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368465784000000
2018-11-20 06:44:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:44:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:44:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:44:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:44:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368465784000000
2018-11-20 06:44:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 06:44:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:44:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:44:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:44:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:44:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:44:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:44:12 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:44:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 06:44:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:44:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:44:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:44:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:44:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:44:12 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:44:12 [DEBUG   ] Finished updating board flags in 1.583 seconds
2018-11-20 06:44:25 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 06:44:45 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 06:45:05 [DEBUG   ] Finished updating ant stats in 1.011 seconds
2018-11-20 06:45:09 [DEBUG   ] Got command RPT from MCS: ref #566450
2018-11-20 06:45:09 [DEBUG   ] summary = ERROR
2018-11-20 06:45:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566450  13 58442 24309793 A  ERRORERROR'
2018-11-20 06:45:10 [DEBUG   ] Got command RPT from MCS: ref #566451
2018-11-20 06:45:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:45:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566451  71 58442 24310001 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:45:10 [DEBUG   ] Got command RPT from MCS: ref #566452
2018-11-20 06:45:10 [DEBUG   ] summary = ERROR
2018-11-20 06:45:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566452  13 58442 24310193 A  ERRORERROR'
2018-11-20 06:45:10 [DEBUG   ] Got command RPT from MCS: ref #566453
2018-11-20 06:45:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:45:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566453  71 58442 24310401 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:45:10 [DEBUG   ] Got command RPT from MCS: ref #566454
2018-11-20 06:45:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:45:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566454  12 58442 24310605 A  ERRORB¢ff'
2018-11-20 06:45:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:45:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368477544000000
2018-11-20 06:45:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:45:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:45:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:45:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:45:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368477544000000
2018-11-20 06:45:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:45:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:45:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:45:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:45:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:45:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:45:12 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:45:12 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:45:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:45:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:45:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:45:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:45:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:45:12 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:45:12 [DEBUG   ] Finished updating board flags in 1.436 seconds
2018-11-20 06:45:25 [DEBUG   ] Finished updating ant stats in 0.991 seconds
2018-11-20 06:45:45 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 06:45:46 [DEBUG   ] Got command RPT from MCS: ref #566455
2018-11-20 06:45:46 [DEBUG   ] summary = ERROR
2018-11-20 06:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566455  13 58442 24346629 A  ERRORERROR'
2018-11-20 06:45:46 [DEBUG   ] Got command RPT from MCS: ref #566456
2018-11-20 06:45:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566456  71 58442 24346630 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:46:05 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-20 06:46:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:46:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368489304000000
2018-11-20 06:46:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:46:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:46:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:46:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:46:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368489304000000
2018-11-20 06:46:11 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:46:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:46:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:46:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:46:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:46:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:46:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:46:12 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:46:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:46:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:46:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:46:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:46:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:46:12 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:46:12 [DEBUG   ] Finished updating board flags in 1.375 seconds
2018-11-20 06:46:25 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 06:46:45 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 06:47:05 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 06:47:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:47:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368501064000000
2018-11-20 06:47:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:47:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:47:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:47:11 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:47:11 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368501064000000
2018-11-20 06:47:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 06:47:11 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 06:47:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:47:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:47:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302368501260000000
2018-11-20 06:47:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:47:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:47:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:47:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:47:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302368501260000000
2018-11-20 06:47:12 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:47:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:47:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:47:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:47:12 [WARNING ] (13, 1, 'bot')
2018-11-20 06:47:12 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:47:12 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 06:47:12 [DEBUG   ] Finished updating board flags in 1.366 seconds
2018-11-20 06:47:26 [DEBUG   ] Finished updating ant stats in 1.396 seconds
2018-11-20 06:47:46 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 06:48:06 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 06:48:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:48:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:48:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:48:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:48:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:48:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:48:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:48:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:48:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:48:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:48:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:48:13 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:48:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:48:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:48:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:48:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:48:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:48:13 [DEBUG   ] Finished updating board flags in 2.043 seconds
2018-11-20 06:48:26 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 06:48:46 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 06:49:06 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 06:49:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:49:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:49:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:49:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:49:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:49:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:49:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:49:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:49:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:49:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:49:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:49:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:49:13 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:49:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:49:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:49:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:49:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:49:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:49:13 [DEBUG   ] Finished updating board flags in 2.009 seconds
2018-11-20 06:49:26 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 06:49:46 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 06:50:06 [DEBUG   ] Finished updating ant stats in 1.010 seconds
2018-11-20 06:50:09 [DEBUG   ] Got command RPT from MCS: ref #566482
2018-11-20 06:50:09 [DEBUG   ] summary = ERROR
2018-11-20 06:50:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566482  13 58442 24609873 A  ERRORERROR'
2018-11-20 06:50:10 [DEBUG   ] Got command RPT from MCS: ref #566483
2018-11-20 06:50:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:50:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566483  98 58442 24610085 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:50:10 [DEBUG   ] Got command RPT from MCS: ref #566484
2018-11-20 06:50:10 [DEBUG   ] summary = ERROR
2018-11-20 06:50:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566484  13 58442 24610277 A  ERRORERROR'
2018-11-20 06:50:10 [DEBUG   ] Got command RPT from MCS: ref #566486
2018-11-20 06:50:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:50:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566486  98 58442 24610485 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:50:10 [DEBUG   ] Got command RPT from MCS: ref #566487
2018-11-20 06:50:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:50:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566487  12 58442 24610677 A  ERRORB¢ff'
2018-11-20 06:50:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:50:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:50:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:50:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:50:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:50:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 06:50:12 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 06:50:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:50:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:50:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:50:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:50:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:50:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:50:13 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:50:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 06:50:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:50:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:50:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:50:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:50:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:50:13 [DEBUG   ] Finished updating board flags in 2.025 seconds
2018-11-20 06:50:26 [DEBUG   ] Finished updating ant stats in 0.990 seconds
2018-11-20 06:50:46 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 06:50:46 [DEBUG   ] Got command RPT from MCS: ref #566492
2018-11-20 06:50:46 [DEBUG   ] summary = ERROR
2018-11-20 06:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566492  13 58442 24646633 A  ERRORERROR'
2018-11-20 06:50:46 [DEBUG   ] Got command RPT from MCS: ref #566493
2018-11-20 06:50:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 06:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566493  98 58442 24646634 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 06:51:06 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 06:51:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:51:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:51:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:51:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:51:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:51:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:51:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:51:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:51:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:51:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:51:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:51:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:51:13 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:51:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:51:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:51:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:51:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:51:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:51:13 [DEBUG   ] Finished updating board flags in 1.966 seconds
2018-11-20 06:51:26 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 06:51:46 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 06:52:06 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 06:52:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:52:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:52:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:52:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:52:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:52:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 06:52:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:52:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:52:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:52:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:52:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:52:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:52:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:52:13 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 06:52:13 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:52:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:52:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:52:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:52:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:52:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:52:13 [DEBUG   ] Finished updating board flags in 1.815 seconds
2018-11-20 06:52:26 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 06:52:46 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 06:53:06 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 06:53:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:53:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:53:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:53:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:53:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:53:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:53:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:53:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:53:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:53:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:53:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:53:13 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:53:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:53:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:53:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:53:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:53:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:53:13 [DEBUG   ] Finished updating board flags in 1.780 seconds
2018-11-20 06:53:27 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 06:53:46 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-20 06:54:07 [DEBUG   ] Finished updating ant stats in 1.395 seconds
2018-11-20 06:54:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:54:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368583580000000
2018-11-20 06:54:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:54:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:54:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:54:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:54:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368583580000000
2018-11-20 06:54:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:54:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:54:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:54:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:54:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:54:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:54:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:54:13 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:54:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:54:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:54:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:54:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:54:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:54:13 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:54:13 [DEBUG   ] Finished updating board flags in 1.620 seconds
2018-11-20 06:54:27 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 06:54:47 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 06:55:07 [DEBUG   ] Finished updating ant stats in 1.214 seconds
2018-11-20 06:55:09 [DEBUG   ] Got command RPT from MCS: ref #566513
2018-11-20 06:55:09 [DEBUG   ] summary = ERROR
2018-11-20 06:55:09 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566513  13 58442 24909954 A  ERRORERROR'
2018-11-20 06:55:10 [DEBUG   ] Got command RPT from MCS: ref #566514
2018-11-20 06:55:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:55:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566514  71 58442 24910161 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:55:10 [DEBUG   ] Got command RPT from MCS: ref #566515
2018-11-20 06:55:10 [DEBUG   ] summary = ERROR
2018-11-20 06:55:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566515  13 58442 24910361 A  ERRORERROR'
2018-11-20 06:55:10 [DEBUG   ] Got command RPT from MCS: ref #566516
2018-11-20 06:55:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:55:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566516  71 58442 24910557 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:55:10 [DEBUG   ] Got command RPT from MCS: ref #566517
2018-11-20 06:55:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 06:55:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566517  12 58442 24910765 A  ERRORB¡ff'
2018-11-20 06:55:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:55:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368595340000000
2018-11-20 06:55:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:55:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:55:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:55:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:55:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368595340000000
2018-11-20 06:55:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 06:55:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:55:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:55:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:55:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:55:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:55:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:55:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:55:13 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:55:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:55:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:55:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:55:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:55:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:55:13 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:55:13 [DEBUG   ] Finished updating board flags in 1.607 seconds
2018-11-20 06:55:27 [DEBUG   ] Finished updating ant stats in 1.187 seconds
2018-11-20 06:55:46 [DEBUG   ] Got command RPT from MCS: ref #566522
2018-11-20 06:55:46 [DEBUG   ] summary = ERROR
2018-11-20 06:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566522  13 58442 24946637 A  ERRORERROR'
2018-11-20 06:55:46 [DEBUG   ] Got command RPT from MCS: ref #566523
2018-11-20 06:55:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 06:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566523  71 58442 24946638 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 06:55:47 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 06:56:07 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 06:56:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:56:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368607100000000
2018-11-20 06:56:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:56:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:56:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:56:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:56:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368607100000000
2018-11-20 06:56:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 06:56:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:56:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:56:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:56:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:56:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:56:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:56:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:56:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 06:56:13 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:56:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:56:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:56:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:56:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:56:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:56:13 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:56:13 [DEBUG   ] Finished updating board flags in 1.584 seconds
2018-11-20 06:56:27 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 06:56:47 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 06:57:07 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 06:57:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:57:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368618860000000
2018-11-20 06:57:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:57:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:57:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:57:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:57:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368618860000000
2018-11-20 06:57:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:57:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 06:57:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:57:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:57:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:57:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:57:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:57:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:57:13 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:57:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:57:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:57:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:57:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:57:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:57:13 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:57:13 [DEBUG   ] Finished updating board flags in 1.400 seconds
2018-11-20 06:57:27 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 06:57:47 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 06:58:07 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 06:58:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:58:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368630620000000
2018-11-20 06:58:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:58:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:58:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:58:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:58:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368630620000000
2018-11-20 06:58:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:58:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:58:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:58:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:58:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:58:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:58:13 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 06:58:13 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 06:58:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 06:58:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:58:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:58:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:58:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:58:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:58:13 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:58:13 [DEBUG   ] Finished updating board flags in 1.420 seconds
2018-11-20 06:58:27 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 06:58:47 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 06:59:07 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 06:59:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 06:59:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368642380000000
2018-11-20 06:59:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 06:59:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 06:59:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 06:59:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 06:59:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368642380000000
2018-11-20 06:59:12 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 06:59:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 06:59:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 06:59:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 06:59:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 06:59:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 06:59:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 06:59:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 06:59:13 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 06:59:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:59:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:59:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:59:13 [WARNING ] (13, 1, 'bot')
2018-11-20 06:59:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 06:59:13 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 06:59:13 [DEBUG   ] Finished updating board flags in 1.410 seconds
2018-11-20 06:59:28 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 06:59:48 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 07:00:08 [DEBUG   ] Finished updating ant stats in 1.216 seconds
2018-11-20 07:00:10 [DEBUG   ] Got command RPT from MCS: ref #566545
2018-11-20 07:00:10 [DEBUG   ] summary = ERROR
2018-11-20 07:00:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566545  13 58442 25210033 A  ERRORERROR'
2018-11-20 07:00:10 [DEBUG   ] Got command RPT from MCS: ref #566546
2018-11-20 07:00:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:00:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566546  71 58442 25210241 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:00:10 [DEBUG   ] Got command RPT from MCS: ref #566547
2018-11-20 07:00:10 [DEBUG   ] summary = ERROR
2018-11-20 07:00:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566547  13 58442 25210445 A  ERRORERROR'
2018-11-20 07:00:10 [DEBUG   ] Got command RPT from MCS: ref #566549
2018-11-20 07:00:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:00:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566549  71 58442 25210637 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:00:10 [DEBUG   ] Got command RPT from MCS: ref #566550
2018-11-20 07:00:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:00:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566550  12 58442 25210841 A  ERRORB ff'
2018-11-20 07:00:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:00:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368654140000000
2018-11-20 07:00:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:00:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:00:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:00:12 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:00:12 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368654140000000
2018-11-20 07:00:12 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:00:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:00:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:00:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:00:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:00:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:00:13 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:00:13 [WARNING ] (13, 1, 'bot')
2018-11-20 07:00:13 [WARNING ] (13, 1, 'bot')
2018-11-20 07:00:13 [WARNING ] (13, 1, 'bot')
2018-11-20 07:00:13 [WARNING ] (13, 1, 'bot')
2018-11-20 07:00:13 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:00:13 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:00:13 [DEBUG   ] Finished updating board flags in 1.386 seconds
2018-11-20 07:00:28 [DEBUG   ] Finished updating ant stats in 1.182 seconds
2018-11-20 07:00:46 [DEBUG   ] Got command RPT from MCS: ref #566555
2018-11-20 07:00:46 [DEBUG   ] summary = ERROR
2018-11-20 07:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566555  13 58442 25246629 A  ERRORERROR'
2018-11-20 07:00:46 [DEBUG   ] Got command RPT from MCS: ref #566556
2018-11-20 07:00:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566556  71 58442 25246630 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:00:48 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 07:01:08 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 07:01:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:01:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:01:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:01:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:01:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:01:13 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:01:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:01:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:01:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:01:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:01:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:01:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:01:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 07:01:14 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:01:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:01:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:01:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:01:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:01:14 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:01:14 [DEBUG   ] Finished updating board flags in 2.201 seconds
2018-11-20 07:01:28 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 07:01:48 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 07:02:08 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 07:02:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:02:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:02:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:02:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:02:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:02:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 07:02:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:02:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:02:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:02:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:02:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:02:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:02:14 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:02:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:02:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:02:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:02:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:02:14 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:02:14 [DEBUG   ] Finished updating board flags in 2.186 seconds
2018-11-20 07:02:28 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 07:02:48 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 07:03:08 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 07:03:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:03:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:03:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:03:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:03:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:03:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:03:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:03:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:03:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:03:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:03:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:03:14 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:03:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:03:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:03:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:03:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:03:14 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:03:14 [DEBUG   ] Finished updating board flags in 2.017 seconds
2018-11-20 07:03:28 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:03:48 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 07:04:08 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 07:04:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:04:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:04:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:04:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:04:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:04:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 07:04:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:04:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 07:04:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:04:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:04:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:04:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:04:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:04:14 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:04:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 07:04:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:04:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:04:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:04:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:04:14 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:04:14 [DEBUG   ] Finished updating board flags in 1.986 seconds
2018-11-20 07:04:28 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 07:04:49 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 07:05:08 [DEBUG   ] Finished updating ant stats in 0.411 seconds
2018-11-20 07:05:10 [DEBUG   ] Got command RPT from MCS: ref #566580
2018-11-20 07:05:10 [DEBUG   ] summary = ERROR
2018-11-20 07:05:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566580  13 58442 25510125 A  ERRORERROR'
2018-11-20 07:05:10 [DEBUG   ] Got command RPT from MCS: ref #566581
2018-11-20 07:05:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:05:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566581  98 58442 25510321 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:05:10 [DEBUG   ] Got command RPT from MCS: ref #566582
2018-11-20 07:05:10 [DEBUG   ] summary = ERROR
2018-11-20 07:05:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566582  13 58442 25510525 A  ERRORERROR'
2018-11-20 07:05:10 [DEBUG   ] Got command RPT from MCS: ref #566583
2018-11-20 07:05:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:05:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566583  98 58442 25510721 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:05:10 [DEBUG   ] Got command RPT from MCS: ref #566584
2018-11-20 07:05:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:05:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566584  12 58442 25510923 A  ERRORB¢ff'
2018-11-20 07:05:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:05:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:05:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:05:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:05:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:05:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 07:05:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:05:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:05:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:05:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:05:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:05:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:05:15 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:05:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:05:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:05:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:05:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:05:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:05:15 [DEBUG   ] Finished updating board flags in 2.967 seconds
2018-11-20 07:05:29 [DEBUG   ] Finished updating ant stats in 1.389 seconds
2018-11-20 07:05:46 [DEBUG   ] Got command RPT from MCS: ref #566585
2018-11-20 07:05:46 [DEBUG   ] summary = ERROR
2018-11-20 07:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566585  13 58442 25546629 A  ERRORERROR'
2018-11-20 07:05:46 [DEBUG   ] Got command RPT from MCS: ref #566586
2018-11-20 07:05:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566586  98 58442 25546630 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:05:49 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 07:06:09 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 07:06:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:06:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:06:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:06:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:06:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:06:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:06:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 07:06:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:06:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:06:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:06:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:06:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:06:14 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:06:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 07:06:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:06:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:06:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:06:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:06:14 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:06:14 [DEBUG   ] Finished updating board flags in 1.646 seconds
2018-11-20 07:06:29 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 07:06:49 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 07:07:09 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 07:07:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:07:13 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368736656000000
2018-11-20 07:07:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:07:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:07:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:07:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:07:13 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368736656000000
2018-11-20 07:07:13 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 07:07:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:07:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:07:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:07:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:07:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:07:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:07:14 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:07:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 07:07:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:07:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:07:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:07:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:07:14 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:07:14 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:07:14 [DEBUG   ] Finished updating board flags in 1.750 seconds
2018-11-20 07:07:29 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 07:07:49 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 07:08:09 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 07:08:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:08:13 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368748416000000
2018-11-20 07:08:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:08:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:08:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:08:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:08:13 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368748416000000
2018-11-20 07:08:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:08:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:08:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:08:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:08:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:08:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:08:14 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:08:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa7, chip 5, bottom port
2018-11-20 07:08:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:08:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:08:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:08:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:08:14 [WARNING ] (25, 5, 'bot')
2018-11-20 07:08:14 [WARNING ] (25, 5, 'bot')
2018-11-20 07:08:14 [WARNING ] (25, 5, 'bot')
2018-11-20 07:08:14 [WARNING ] (25, 5, 'bot')
2018-11-20 07:08:14 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:08:14 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:08:14 [DEBUG   ] Finished updating board flags in 1.443 seconds
2018-11-20 07:08:29 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 07:08:49 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:09:10 [DEBUG   ] Finished updating ant stats in 1.394 seconds
2018-11-20 07:09:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:09:13 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368760176000000
2018-11-20 07:09:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:09:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:09:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:09:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:09:13 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368760176000000
2018-11-20 07:09:13 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:09:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:09:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:09:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:09:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:09:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:09:14 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:09:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:09:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:09:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:09:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:09:14 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:09:14 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:09:14 [DEBUG   ] Finished updating board flags in 1.588 seconds
2018-11-20 07:09:30 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 07:09:50 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 07:10:10 [DEBUG   ] Got command RPT from MCS: ref #566608
2018-11-20 07:10:10 [DEBUG   ] summary = ERROR
2018-11-20 07:10:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566608  13 58442 25810201 A  ERRORERROR'
2018-11-20 07:10:10 [DEBUG   ] Finished updating ant stats in 1.007 seconds
2018-11-20 07:10:10 [DEBUG   ] Got command RPT from MCS: ref #566609
2018-11-20 07:10:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:10:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566609  71 58442 25810401 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:10:10 [DEBUG   ] Got command RPT from MCS: ref #566610
2018-11-20 07:10:10 [DEBUG   ] summary = ERROR
2018-11-20 07:10:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566610  13 58442 25810597 A  ERRORERROR'
2018-11-20 07:10:10 [DEBUG   ] Got command RPT from MCS: ref #566612
2018-11-20 07:10:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:10:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566612  71 58442 25810801 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:10:10 [DEBUG   ] Got command RPT from MCS: ref #566613
2018-11-20 07:10:10 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:10:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566613  12 58442 25810997 A  ERRORB¢ff'
2018-11-20 07:10:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:10:13 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368771936000000
2018-11-20 07:10:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:10:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:10:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:10:13 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:10:13 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368771936000000
2018-11-20 07:10:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:10:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:10:14 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302368772132000000
2018-11-20 07:10:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:10:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:10:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:10:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:10:14 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302368772132000000
2018-11-20 07:10:14 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:10:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:10:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:10:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:10:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:10:14 [WARNING ] (13, 1, 'bot')
2018-11-20 07:10:14 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:10:14 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 07:10:14 [DEBUG   ] Finished updating board flags in 1.388 seconds
2018-11-20 07:10:31 [DEBUG   ] Finished updating ant stats in 1.997 seconds
2018-11-20 07:10:46 [DEBUG   ] Got command RPT from MCS: ref #566622
2018-11-20 07:10:46 [DEBUG   ] summary = ERROR
2018-11-20 07:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566622  13 58442 25846637 A  ERRORERROR'
2018-11-20 07:10:46 [DEBUG   ] Got command RPT from MCS: ref #566623
2018-11-20 07:10:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 07:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566623  79 58442 25846638 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 07:10:50 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 07:11:10 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 07:11:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:11:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:11:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:11:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:11:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:11:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 07:11:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:11:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:11:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:11:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:11:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:11:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:11:15 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:11:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:11:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:11:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:11:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:11:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:11:15 [DEBUG   ] Finished updating board flags in 2.048 seconds
2018-11-20 07:11:30 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 07:11:50 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 07:12:10 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 07:12:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:12:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:12:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:12:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:12:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:12:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:12:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:12:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:12:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:12:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:12:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:12:15 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:12:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:12:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:12:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:12:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:12:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:12:15 [DEBUG   ] Finished updating board flags in 2.008 seconds
2018-11-20 07:12:30 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:12:51 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-20 07:13:11 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 07:13:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:13:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:13:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:13:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:13:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:13:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:13:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:13:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:13:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:13:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:13:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:13:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 07:13:15 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:13:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:13:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:13:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:13:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:13:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:13:15 [DEBUG   ] Finished updating board flags in 1.992 seconds
2018-11-20 07:13:31 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 07:13:51 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 07:14:11 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 07:14:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:14:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:14:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:14:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:14:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:14:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:14:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:14:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:14:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:14:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:14:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:14:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:14:15 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:14:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 07:14:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:14:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:14:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:14:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:14:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:14:15 [DEBUG   ] Finished updating board flags in 1.810 seconds
2018-11-20 07:14:31 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 07:14:51 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 07:15:10 [DEBUG   ] Got command RPT from MCS: ref #566643
2018-11-20 07:15:10 [DEBUG   ] summary = ERROR
2018-11-20 07:15:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566643  13 58442 26110277 A  ERRORERROR'
2018-11-20 07:15:10 [DEBUG   ] Got command RPT from MCS: ref #566644
2018-11-20 07:15:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:15:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566644  98 58442 26110481 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:15:10 [DEBUG   ] Got command RPT from MCS: ref #566645
2018-11-20 07:15:10 [DEBUG   ] summary = ERROR
2018-11-20 07:15:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566645  13 58442 26110685 A  ERRORERROR'
2018-11-20 07:15:10 [DEBUG   ] Got command RPT from MCS: ref #566646
2018-11-20 07:15:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:15:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566646  98 58442 26110877 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:15:11 [DEBUG   ] Got command RPT from MCS: ref #566647
2018-11-20 07:15:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:15:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566647  12 58442 26111077 A  ERRORB¡ff'
2018-11-20 07:15:11 [DEBUG   ] Finished updating ant stats in 0.816 seconds
2018-11-20 07:15:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:15:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:15:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:15:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:15:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:15:14 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 07:15:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:15:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:15:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:15:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:15:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:15:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:15:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 07:15:15 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:15:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:15:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:15:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:15:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:15:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:15:15 [DEBUG   ] Finished updating board flags in 1.817 seconds
2018-11-20 07:15:31 [DEBUG   ] Finished updating ant stats in 0.788 seconds
2018-11-20 07:15:46 [DEBUG   ] Got command RPT from MCS: ref #566648
2018-11-20 07:15:46 [DEBUG   ] summary = ERROR
2018-11-20 07:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566648  13 58442 26146641 A  ERRORERROR'
2018-11-20 07:15:46 [DEBUG   ] Got command RPT from MCS: ref #566649
2018-11-20 07:15:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566649  98 58442 26146642 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:15:51 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:16:11 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:16:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:16:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:16:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:16:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:16:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:16:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:16:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:16:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:16:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:16:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:16:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:16:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:16:15 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:16:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:16:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:16:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:16:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:16:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:16:15 [DEBUG   ] Finished updating board flags in 1.786 seconds
2018-11-20 07:16:31 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 07:16:52 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 07:17:11 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-20 07:17:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:17:14 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368854452000000
2018-11-20 07:17:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:17:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:17:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:17:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:17:14 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368854452000000
2018-11-20 07:17:14 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:17:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:17:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:17:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:17:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:17:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:17:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:17:15 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:17:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:17:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:17:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:17:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:17:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:17:15 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:17:15 [DEBUG   ] Finished updating board flags in 1.596 seconds
2018-11-20 07:17:32 [DEBUG   ] Finished updating ant stats in 1.394 seconds
2018-11-20 07:17:52 [DEBUG   ] Finished updating ant stats in 1.209 seconds
2018-11-20 07:18:12 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 07:18:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:18:14 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368866212000000
2018-11-20 07:18:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:18:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:18:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:18:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:18:14 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368866212000000
2018-11-20 07:18:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:18:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:18:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:18:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:18:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:18:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:18:15 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:18:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:18:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:18:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:18:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:18:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:18:15 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:18:15 [DEBUG   ] Finished updating board flags in 1.406 seconds
2018-11-20 07:18:32 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 07:18:52 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 07:19:12 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 07:19:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:19:14 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368877972000000
2018-11-20 07:19:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:19:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:19:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:19:14 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:19:14 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368877972000000
2018-11-20 07:19:14 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:19:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:19:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:19:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:19:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:19:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:19:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 07:19:15 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:19:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:19:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:19:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:19:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:19:15 [WARNING ] (13, 1, 'bot')
2018-11-20 07:19:15 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:19:15 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:19:15 [DEBUG   ] Finished updating board flags in 1.392 seconds
2018-11-20 07:19:32 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 07:19:52 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 07:20:10 [DEBUG   ] Got command RPT from MCS: ref #566675
2018-11-20 07:20:10 [DEBUG   ] summary = ERROR
2018-11-20 07:20:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566675  13 58442 26410357 A  ERRORERROR'
2018-11-20 07:20:10 [DEBUG   ] Got command RPT from MCS: ref #566676
2018-11-20 07:20:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:20:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566676  71 58442 26410561 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:20:10 [DEBUG   ] Got command RPT from MCS: ref #566677
2018-11-20 07:20:10 [DEBUG   ] summary = ERROR
2018-11-20 07:20:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566677  13 58442 26410765 A  ERRORERROR'
2018-11-20 07:20:10 [DEBUG   ] Got command RPT from MCS: ref #566678
2018-11-20 07:20:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:20:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566678  71 58442 26410957 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:20:11 [DEBUG   ] Got command RPT from MCS: ref #566680
2018-11-20 07:20:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:20:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566680  12 58442 26411165 A  ERRORB¡ff'
2018-11-20 07:20:12 [DEBUG   ] Finished updating ant stats in 0.814 seconds
2018-11-20 07:20:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:20:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:20:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:20:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:20:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:20:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 4
2018-11-20 07:20:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:20:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:20:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:20:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:20:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:20:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:20:16 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:20:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:20:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:20:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:20:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:20:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:20:16 [DEBUG   ] Finished updating board flags in 2.283 seconds
2018-11-20 07:20:32 [DEBUG   ] Finished updating ant stats in 0.792 seconds
2018-11-20 07:20:46 [DEBUG   ] Got command RPT from MCS: ref #566685
2018-11-20 07:20:46 [DEBUG   ] summary = ERROR
2018-11-20 07:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566685  13 58442 26446641 A  ERRORERROR'
2018-11-20 07:20:46 [DEBUG   ] Got command RPT from MCS: ref #566686
2018-11-20 07:20:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566686  98 58442 26446642 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:20:52 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:21:12 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:21:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:21:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:21:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:21:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:21:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:21:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:21:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:21:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:21:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:21:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:21:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:21:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:21:16 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:21:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:21:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:21:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:21:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:21:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:21:16 [DEBUG   ] Finished updating board flags in 2.117 seconds
2018-11-20 07:21:32 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 07:21:53 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 07:22:13 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 07:22:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:22:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:22:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:22:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:22:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:22:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:22:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:22:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:22:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:22:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:22:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:22:16 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:22:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:22:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:22:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:22:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:22:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:22:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:22:16 [DEBUG   ] Finished updating board flags in 2.018 seconds
2018-11-20 07:22:32 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 07:22:52 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 07:23:13 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 07:23:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:23:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:23:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:23:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:23:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:23:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:23:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:23:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:23:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:23:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:23:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:23:16 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:23:16 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 07:23:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:23:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:23:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:23:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:23:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:23:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:23:16 [DEBUG   ] Finished updating board flags in 2.063 seconds
2018-11-20 07:23:32 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-20 07:23:53 [DEBUG   ] Finished updating ant stats in 1.394 seconds
2018-11-20 07:24:13 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 07:24:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:24:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:24:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:24:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:24:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:24:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 07:24:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 07:24:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:24:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:24:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:24:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:24:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:24:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:24:16 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:24:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:24:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:24:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:24:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:24:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:24:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:24:16 [DEBUG   ] Finished updating board flags in 1.933 seconds
2018-11-20 07:24:33 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-20 07:24:53 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 07:25:10 [DEBUG   ] Got command RPT from MCS: ref #566706
2018-11-20 07:25:10 [DEBUG   ] summary = ERROR
2018-11-20 07:25:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566706  13 58442 26710445 A  ERRORERROR'
2018-11-20 07:25:10 [DEBUG   ] Got command RPT from MCS: ref #566707
2018-11-20 07:25:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:25:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566707  98 58442 26710637 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:25:10 [DEBUG   ] Got command RPT from MCS: ref #566708
2018-11-20 07:25:10 [DEBUG   ] summary = ERROR
2018-11-20 07:25:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566708  13 58442 26710845 A  ERRORERROR'
2018-11-20 07:25:11 [DEBUG   ] Got command RPT from MCS: ref #566709
2018-11-20 07:25:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:25:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566709  98 58442 26711041 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:25:11 [DEBUG   ] Got command RPT from MCS: ref #566710
2018-11-20 07:25:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:25:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566710  12 58442 26711241 A  ERRORB£ff'
2018-11-20 07:25:13 [DEBUG   ] Finished updating ant stats in 1.118 seconds
2018-11-20 07:25:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:25:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:25:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:25:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:25:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:25:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:25:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:25:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:25:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:25:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:25:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:25:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 07:25:16 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:25:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:25:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:25:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:25:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:25:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:25:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:25:16 [DEBUG   ] Finished updating board flags in 1.814 seconds
2018-11-20 07:25:33 [DEBUG   ] Finished updating ant stats in 0.688 seconds
2018-11-20 07:25:46 [DEBUG   ] Got command RPT from MCS: ref #566715
2018-11-20 07:25:46 [DEBUG   ] summary = ERROR
2018-11-20 07:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566715  13 58442 26746642 A  ERRORERROR'
2018-11-20 07:25:46 [DEBUG   ] Got command RPT from MCS: ref #566716
2018-11-20 07:25:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566716  98 58442 26746642 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:25:53 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:26:14 [DEBUG   ] Finished updating ant stats in 1.395 seconds
2018-11-20 07:26:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:26:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:26:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:26:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:26:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:26:15 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 07:26:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:26:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:26:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:26:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:26:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:26:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:26:16 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:26:16 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 07:26:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:26:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:26:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:26:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:26:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:26:16 [DEBUG   ] Finished updating board flags in 1.777 seconds
2018-11-20 07:26:34 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 07:26:54 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 07:27:14 [DEBUG   ] Finished updating ant stats in 1.195 seconds
2018-11-20 07:27:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:27:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368972248000000
2018-11-20 07:27:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:27:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:27:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:27:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:27:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368972248000000
2018-11-20 07:27:15 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:27:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:27:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:27:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:27:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:27:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:27:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:27:16 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:27:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:27:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:27:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:27:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:27:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:27:16 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:27:16 [DEBUG   ] Finished updating board flags in 1.618 seconds
2018-11-20 07:27:34 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 07:27:54 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 07:28:14 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 07:28:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:28:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368984008000000
2018-11-20 07:28:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:28:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:28:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:28:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:28:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368984008000000
2018-11-20 07:28:15 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:28:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:28:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:28:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:28:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:28:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:28:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:28:16 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:28:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 07:28:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:28:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:28:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:28:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:28:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:28:16 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:28:16 [DEBUG   ] Finished updating board flags in 1.608 seconds
2018-11-20 07:28:34 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 07:28:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 07:29:14 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-20 07:29:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:29:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302368995768000000
2018-11-20 07:29:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:29:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:29:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:29:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:29:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302368995768000000
2018-11-20 07:29:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:29:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:29:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:29:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:29:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:29:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:29:16 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:29:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:29:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:29:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:29:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:29:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:29:16 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:29:16 [DEBUG   ] Finished updating board flags in 1.583 seconds
2018-11-20 07:29:34 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 07:29:54 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 07:30:10 [DEBUG   ] Got command RPT from MCS: ref #566738
2018-11-20 07:30:10 [DEBUG   ] summary = ERROR
2018-11-20 07:30:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566738  13 58442 27010522 A  ERRORERROR'
2018-11-20 07:30:10 [DEBUG   ] Got command RPT from MCS: ref #566739
2018-11-20 07:30:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:30:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566739  71 58442 27010717 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:30:10 [DEBUG   ] Got command RPT from MCS: ref #566740
2018-11-20 07:30:10 [DEBUG   ] summary = ERROR
2018-11-20 07:30:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566740  13 58442 27010921 A  ERRORERROR'
2018-11-20 07:30:11 [DEBUG   ] Got command RPT from MCS: ref #566742
2018-11-20 07:30:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:30:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566742  71 58442 27011121 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:30:11 [DEBUG   ] Got command RPT from MCS: ref #566743
2018-11-20 07:30:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:30:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566743  12 58442 27011325 A  ERRORB¢ff'
2018-11-20 07:30:14 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 07:30:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:30:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369007528000000
2018-11-20 07:30:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:30:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:30:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:30:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:30:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369007528000000
2018-11-20 07:30:15 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:30:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:30:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:30:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:30:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:30:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:30:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:30:16 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:30:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:30:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:30:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:30:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:30:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:30:16 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:30:16 [DEBUG   ] Finished updating board flags in 1.433 seconds
2018-11-20 07:30:34 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 07:30:46 [DEBUG   ] Got command RPT from MCS: ref #566748
2018-11-20 07:30:46 [DEBUG   ] summary = ERROR
2018-11-20 07:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566748  13 58442 27046637 A  ERRORERROR'
2018-11-20 07:30:46 [DEBUG   ] Got command RPT from MCS: ref #566749
2018-11-20 07:30:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566749  71 58442 27046638 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:30:54 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:31:14 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 07:31:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:31:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369019288000000
2018-11-20 07:31:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:31:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:31:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:31:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:31:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369019288000000
2018-11-20 07:31:15 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:31:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:31:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:31:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:31:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:31:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:31:16 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:31:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 07:31:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:31:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:31:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:31:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:31:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:31:16 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:31:16 [DEBUG   ] Finished updating board flags in 1.376 seconds
2018-11-20 07:31:34 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 07:31:55 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-20 07:32:15 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 07:32:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:32:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369031048000000
2018-11-20 07:32:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:32:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:32:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:32:15 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:32:15 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369031048000000
2018-11-20 07:32:15 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:32:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:32:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:32:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369031244000000
2018-11-20 07:32:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:32:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:32:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:32:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:32:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369031244000000
2018-11-20 07:32:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:32:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 07:32:16 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:32:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:32:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:32:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:32:16 [WARNING ] (13, 1, 'bot')
2018-11-20 07:32:16 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:32:16 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 07:32:16 [DEBUG   ] Finished updating board flags in 1.388 seconds
2018-11-20 07:32:35 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 07:32:55 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 07:33:15 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 07:33:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:33:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:33:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:33:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:33:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:33:16 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:33:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:33:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:33:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:33:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:33:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:33:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:33:17 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:33:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:33:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:33:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:33:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:33:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:33:17 [DEBUG   ] Finished updating board flags in 2.032 seconds
2018-11-20 07:33:35 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 07:33:55 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 07:34:15 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 07:34:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:34:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:34:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:34:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:34:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:34:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:34:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:34:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:34:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:34:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:34:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:34:17 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:34:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:34:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:34:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:34:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:34:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:34:17 [DEBUG   ] Finished updating board flags in 1.972 seconds
2018-11-20 07:34:35 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 07:34:55 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 07:35:10 [DEBUG   ] Got command RPT from MCS: ref #566773
2018-11-20 07:35:10 [DEBUG   ] summary = ERROR
2018-11-20 07:35:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566773  13 58442 27310597 A  ERRORERROR'
2018-11-20 07:35:10 [DEBUG   ] Got command RPT from MCS: ref #566774
2018-11-20 07:35:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:35:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566774  98 58442 27310802 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:35:11 [DEBUG   ] Got command RPT from MCS: ref #566775
2018-11-20 07:35:11 [DEBUG   ] summary = ERROR
2018-11-20 07:35:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566775  13 58442 27311005 A  ERRORERROR'
2018-11-20 07:35:11 [DEBUG   ] Got command RPT from MCS: ref #566776
2018-11-20 07:35:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:35:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566776  98 58442 27311202 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:35:11 [DEBUG   ] Got command RPT from MCS: ref #566777
2018-11-20 07:35:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:35:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566777  12 58442 27311405 A  ERRORB ff'
2018-11-20 07:35:15 [DEBUG   ] Finished updating ant stats in 1.150 seconds
2018-11-20 07:35:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:35:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:35:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:35:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:35:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:35:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:35:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:35:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:35:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:35:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:35:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:35:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 07:35:17 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:35:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:35:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:35:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:35:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:35:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:35:17 [DEBUG   ] Finished updating board flags in 1.834 seconds
2018-11-20 07:35:35 [DEBUG   ] Finished updating ant stats in 0.648 seconds
2018-11-20 07:35:46 [DEBUG   ] Got command RPT from MCS: ref #566778
2018-11-20 07:35:46 [DEBUG   ] summary = ERROR
2018-11-20 07:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566778  13 58442 27346646 A  ERRORERROR'
2018-11-20 07:35:46 [DEBUG   ] Got command RPT from MCS: ref #566779
2018-11-20 07:35:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566779  98 58442 27346646 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:35:55 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:36:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:36:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:36:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:36:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:36:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:36:16 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 07:36:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:36:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:36:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:36:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:36:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:36:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:36:17 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:36:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:36:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:36:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:36:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:36:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:36:17 [DEBUG   ] Finished updating board flags in 1.775 seconds
2018-11-20 07:36:36 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 07:36:56 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 07:37:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:37:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369090044000000
2018-11-20 07:37:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:37:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:37:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:37:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:37:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369090044000000
2018-11-20 07:37:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:37:16 [DEBUG   ] Finished updating ant stats in 1.049 seconds
2018-11-20 07:37:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 07:37:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:37:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:37:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:37:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:37:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:37:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:37:17 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:37:17 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:37:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:37:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:37:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:37:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:37:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:37:17 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:37:17 [DEBUG   ] Finished updating board flags in 1.618 seconds
2018-11-20 07:37:36 [DEBUG   ] Finished updating ant stats in 0.956 seconds
2018-11-20 07:37:56 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 07:38:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:38:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369101804000000
2018-11-20 07:38:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:38:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:38:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:38:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:38:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369101804000000
2018-11-20 07:38:16 [DEBUG   ] Finished updating ant stats in 0.847 seconds
2018-11-20 07:38:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:38:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 07:38:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:38:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:38:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:38:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:38:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:38:17 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:38:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:38:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:38:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:38:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:38:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:38:17 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:38:17 [DEBUG   ] Finished updating board flags in 1.580 seconds
2018-11-20 07:38:36 [DEBUG   ] Finished updating ant stats in 0.754 seconds
2018-11-20 07:38:56 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 07:39:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:39:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369113564000000
2018-11-20 07:39:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:39:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:39:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:39:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:39:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369113564000000
2018-11-20 07:39:16 [DEBUG   ] Finished updating ant stats in 0.646 seconds
2018-11-20 07:39:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 07:39:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:39:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:39:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:39:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:39:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:39:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:39:17 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:39:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:39:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:39:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:39:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:39:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:39:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:39:17 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:39:17 [DEBUG   ] Finished updating board flags in 1.411 seconds
2018-11-20 07:39:36 [DEBUG   ] Finished updating ant stats in 0.555 seconds
2018-11-20 07:39:57 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 07:40:10 [DEBUG   ] Got command RPT from MCS: ref #566801
2018-11-20 07:40:10 [DEBUG   ] summary = ERROR
2018-11-20 07:40:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566801  13 58442 27610681 A  ERRORERROR'
2018-11-20 07:40:10 [DEBUG   ] Got command RPT from MCS: ref #566802
2018-11-20 07:40:10 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:40:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566802  71 58442 27610885 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:40:11 [DEBUG   ] Got command RPT from MCS: ref #566803
2018-11-20 07:40:11 [DEBUG   ] summary = ERROR
2018-11-20 07:40:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566803  13 58442 27611077 A  ERRORERROR'
2018-11-20 07:40:11 [DEBUG   ] Got command RPT from MCS: ref #566805
2018-11-20 07:40:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:40:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566805  71 58442 27611281 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:40:11 [DEBUG   ] Got command RPT from MCS: ref #566807
2018-11-20 07:40:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:40:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566807  12 58442 27611485 A  ERRORB¢ff'
2018-11-20 07:40:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:40:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369125324000000
2018-11-20 07:40:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:40:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:40:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:40:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:40:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369125324000000
2018-11-20 07:40:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:40:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 07:40:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:40:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:40:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:40:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:40:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:40:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:40:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-20 07:40:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:40:17 [DEBUG   ] Finished updating ant stats in 1.252 seconds
2018-11-20 07:40:17 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:40:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:40:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:40:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:40:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:40:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:40:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:40:17 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:40:17 [DEBUG   ] Finished updating board flags in 1.457 seconds
2018-11-20 07:40:37 [DEBUG   ] Finished updating ant stats in 1.156 seconds
2018-11-20 07:40:46 [DEBUG   ] Got command RPT from MCS: ref #566815
2018-11-20 07:40:46 [DEBUG   ] summary = ERROR
2018-11-20 07:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566815  13 58442 27646641 A  ERRORERROR'
2018-11-20 07:40:46 [DEBUG   ] Got command RPT from MCS: ref #566816
2018-11-20 07:40:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566816  71 58442 27646642 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:40:57 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 07:41:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:41:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369137084000000
2018-11-20 07:41:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:41:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:41:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:41:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:41:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369137084000000
2018-11-20 07:41:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 07:41:16 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:41:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:41:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:41:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:41:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:41:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:41:17 [DEBUG   ] Finished updating ant stats in 1.046 seconds
2018-11-20 07:41:17 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:41:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:41:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:41:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:41:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:41:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:41:17 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:41:17 [DEBUG   ] Finished updating board flags in 1.433 seconds
2018-11-20 07:41:37 [DEBUG   ] Finished updating ant stats in 0.960 seconds
2018-11-20 07:41:57 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 07:42:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:42:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369148844000000
2018-11-20 07:42:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:42:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:42:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:42:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:42:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369148844000000
2018-11-20 07:42:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:42:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:42:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369149040000000
2018-11-20 07:42:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:42:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:42:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:42:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:42:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369149040000000
2018-11-20 07:42:17 [DEBUG   ] Finished updating ant stats in 0.643 seconds
2018-11-20 07:42:17 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:42:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 07:42:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:42:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:42:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:42:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:42:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:42:17 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 07:42:17 [DEBUG   ] Finished updating board flags in 1.458 seconds
2018-11-20 07:42:37 [DEBUG   ] Finished updating ant stats in 0.560 seconds
2018-11-20 07:42:58 [DEBUG   ] Finished updating ant stats in 1.394 seconds
2018-11-20 07:43:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:43:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369160604000000
2018-11-20 07:43:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:43:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:43:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:43:16 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:43:16 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369160604000000
2018-11-20 07:43:16 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:43:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:43:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:43:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369160800000000
2018-11-20 07:43:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:43:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:43:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:43:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:43:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369160800000000
2018-11-20 07:43:17 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:43:17 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:43:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:43:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:43:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:43:17 [WARNING ] (13, 1, 'bot')
2018-11-20 07:43:17 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:43:17 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 07:43:17 [DEBUG   ] Finished updating board flags in 1.494 seconds
2018-11-20 07:43:18 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 07:43:38 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 07:43:58 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 07:44:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:44:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:44:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:44:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:44:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:44:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:44:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 07:44:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:44:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:44:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:44:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:44:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:44:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:44:18 [DEBUG   ] Finished updating ant stats in 1.242 seconds
2018-11-20 07:44:18 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:44:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:44:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:44:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:44:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:44:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:44:18 [DEBUG   ] Finished updating board flags in 2.229 seconds
2018-11-20 07:44:38 [DEBUG   ] Finished updating ant stats in 1.158 seconds
2018-11-20 07:44:58 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 07:45:10 [DEBUG   ] Got command RPT from MCS: ref #566836
2018-11-20 07:45:10 [DEBUG   ] summary = ERROR
2018-11-20 07:45:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566836  13 58442 27910757 A  ERRORERROR'
2018-11-20 07:45:10 [DEBUG   ] Got command RPT from MCS: ref #566837
2018-11-20 07:45:10 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:45:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566837  98 58442 27910961 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:45:11 [DEBUG   ] Got command RPT from MCS: ref #566838
2018-11-20 07:45:11 [DEBUG   ] summary = ERROR
2018-11-20 07:45:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566838  13 58442 27911165 A  ERRORERROR'
2018-11-20 07:45:11 [DEBUG   ] Got command RPT from MCS: ref #566839
2018-11-20 07:45:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:45:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566839  98 58442 27911357 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:45:11 [DEBUG   ] Got command RPT from MCS: ref #566840
2018-11-20 07:45:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:45:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566840  12 58442 27911561 A  ERRORB£ff'
2018-11-20 07:45:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:45:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:45:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:45:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:45:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:45:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:45:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:45:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:45:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:45:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:45:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:45:18 [DEBUG   ] Finished updating ant stats in 1.045 seconds
2018-11-20 07:45:18 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:45:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:45:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:45:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:45:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:45:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:45:18 [DEBUG   ] Finished updating board flags in 2.184 seconds
2018-11-20 07:45:38 [DEBUG   ] Finished updating ant stats in 0.954 seconds
2018-11-20 07:45:46 [DEBUG   ] Got command RPT from MCS: ref #566841
2018-11-20 07:45:46 [DEBUG   ] summary = ERROR
2018-11-20 07:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566841  13 58442 27946641 A  ERRORERROR'
2018-11-20 07:45:46 [DEBUG   ] Got command RPT from MCS: ref #566842
2018-11-20 07:45:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566842  98 58442 27946642 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:45:58 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 07:46:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:46:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:46:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:46:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:46:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:46:17 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:46:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:46:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:46:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:46:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:46:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:46:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:46:18 [DEBUG   ] Finished updating ant stats in 0.848 seconds
2018-11-20 07:46:18 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:46:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:46:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:46:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:46:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:46:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:46:18 [DEBUG   ] Finished updating board flags in 2.014 seconds
2018-11-20 07:46:38 [DEBUG   ] Finished updating ant stats in 0.756 seconds
2018-11-20 07:46:58 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 07:47:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:47:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:47:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:47:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:47:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:47:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:47:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:47:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:47:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:47:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:47:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:47:18 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:47:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:47:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:47:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:47:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:47:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:47:18 [DEBUG   ] Finished updating board flags in 1.993 seconds
2018-11-20 07:47:19 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 07:47:38 [DEBUG   ] Finished updating ant stats in 0.405 seconds
2018-11-20 07:47:59 [DEBUG   ] Finished updating ant stats in 1.392 seconds
2018-11-20 07:48:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:48:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:48:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:48:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:48:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:48:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 07:48:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:48:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:48:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:48:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:48:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:48:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:48:18 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:48:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:48:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:48:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:48:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:48:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:48:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:48:18 [DEBUG   ] Finished updating board flags in 1.835 seconds
2018-11-20 07:48:19 [DEBUG   ] Finished updating ant stats in 1.210 seconds
2018-11-20 07:48:39 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 07:48:59 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 07:49:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:49:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:49:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:49:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:49:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:49:17 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:49:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 07:49:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:49:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:49:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:49:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:49:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:49:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:49:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 07:49:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:49:18 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:49:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 07:49:18 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 07:49:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:49:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:49:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:49:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:49:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:49:18 [DEBUG   ] Finished updating board flags in 1.766 seconds
2018-11-20 07:49:19 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 07:49:39 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 07:49:59 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 07:50:10 [DEBUG   ] Got command RPT from MCS: ref #566868
2018-11-20 07:50:10 [DEBUG   ] summary = ERROR
2018-11-20 07:50:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566868  13 58442 28210837 A  ERRORERROR'
2018-11-20 07:50:11 [DEBUG   ] Got command RPT from MCS: ref #566869
2018-11-20 07:50:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:50:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566869  98 58442 28211041 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:50:11 [DEBUG   ] Got command RPT from MCS: ref #566870
2018-11-20 07:50:11 [DEBUG   ] summary = ERROR
2018-11-20 07:50:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566870  13 58442 28211233 A  ERRORERROR'
2018-11-20 07:50:11 [DEBUG   ] Got command RPT from MCS: ref #566872
2018-11-20 07:50:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:50:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566872  98 58442 28211437 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:50:11 [DEBUG   ] Got command RPT from MCS: ref #566874
2018-11-20 07:50:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:50:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566874  12 58442 28211641 A  ERRORB¢ff'
2018-11-20 07:50:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:50:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369243120000000
2018-11-20 07:50:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:50:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:50:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:50:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:50:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369243120000000
2018-11-20 07:50:17 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:50:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:50:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:50:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:50:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:50:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:50:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:50:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 07:50:18 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:50:18 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:50:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:50:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:50:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:50:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:50:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:50:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:50:18 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:50:18 [DEBUG   ] Finished updating board flags in 1.632 seconds
2018-11-20 07:50:19 [DEBUG   ] Finished updating ant stats in 0.994 seconds
2018-11-20 07:50:39 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 07:50:46 [DEBUG   ] Got command RPT from MCS: ref #566878
2018-11-20 07:50:46 [DEBUG   ] summary = ERROR
2018-11-20 07:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566878  13 58442 28246649 A  ERRORERROR'
2018-11-20 07:50:46 [DEBUG   ] Got command RPT from MCS: ref #566879
2018-11-20 07:50:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 07:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566879  71 58442 28246650 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 07:50:59 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 07:51:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:51:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369254880000000
2018-11-20 07:51:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:51:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:51:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:51:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:51:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369254880000000
2018-11-20 07:51:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:51:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:51:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:51:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:51:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:51:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:51:18 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:51:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 07:51:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:51:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:51:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:51:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:51:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:51:18 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:51:18 [DEBUG   ] Finished updating board flags in 1.593 seconds
2018-11-20 07:51:19 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 07:51:39 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 07:51:59 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 07:52:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:52:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369266640000000
2018-11-20 07:52:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:52:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:52:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:52:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:52:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369266640000000
2018-11-20 07:52:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:52:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 07:52:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:52:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:52:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:52:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:52:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:52:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:52:18 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:52:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:52:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:52:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:52:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:52:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:52:18 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 07:52:18 [DEBUG   ] Finished updating board flags in 1.433 seconds
2018-11-20 07:52:19 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 07:52:39 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 07:52:59 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:53:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:53:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369278400000000
2018-11-20 07:53:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:53:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:53:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:53:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:53:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369278400000000
2018-11-20 07:53:17 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:53:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:53:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369278596000000
2018-11-20 07:53:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:53:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:53:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:53:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:53:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369278596000000
2018-11-20 07:53:18 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:53:18 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 07:53:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:53:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:53:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:53:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:53:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:53:18 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 07:53:18 [DEBUG   ] Finished updating board flags in 1.464 seconds
2018-11-20 07:53:19 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 07:53:40 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 07:54:00 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-20 07:54:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:54:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369290160000000
2018-11-20 07:54:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:54:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:54:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:54:17 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:54:17 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369290160000000
2018-11-20 07:54:17 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 07:54:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:54:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:54:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369290356000000
2018-11-20 07:54:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:54:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:54:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:54:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:54:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369290356000000
2018-11-20 07:54:18 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:54:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:54:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:54:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:54:18 [WARNING ] (13, 1, 'bot')
2018-11-20 07:54:18 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:54:18 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 07:54:18 [DEBUG   ] Finished updating board flags in 1.457 seconds
2018-11-20 07:54:20 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 07:54:40 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 07:55:00 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 07:55:10 [DEBUG   ] Got command RPT from MCS: ref #566899
2018-11-20 07:55:10 [DEBUG   ] summary = ERROR
2018-11-20 07:55:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566899  13 58442 28510917 A  ERRORERROR'
2018-11-20 07:55:11 [DEBUG   ] Got command RPT from MCS: ref #566900
2018-11-20 07:55:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 07:55:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566900  79 58442 28511121 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 07:55:11 [DEBUG   ] Got command RPT from MCS: ref #566901
2018-11-20 07:55:11 [DEBUG   ] summary = ERROR
2018-11-20 07:55:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566901  13 58442 28511313 A  ERRORERROR'
2018-11-20 07:55:11 [DEBUG   ] Got command RPT from MCS: ref #566902
2018-11-20 07:55:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 07:55:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566902  79 58442 28511517 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 07:55:11 [DEBUG   ] Got command RPT from MCS: ref #566903
2018-11-20 07:55:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 07:55:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566903  12 58442 28511721 A  ERRORB¡ff'
2018-11-20 07:55:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:55:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:55:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:55:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:55:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:55:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 07:55:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:55:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:55:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:55:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:55:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:55:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:55:19 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:55:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 07:55:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:55:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:55:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:55:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:55:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:55:19 [DEBUG   ] Finished updating board flags in 2.245 seconds
2018-11-20 07:55:20 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 07:55:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 07:55:46 [DEBUG   ] Got command RPT from MCS: ref #566908
2018-11-20 07:55:46 [DEBUG   ] summary = ERROR
2018-11-20 07:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566908  13 58442 28546645 A  ERRORERROR'
2018-11-20 07:55:46 [DEBUG   ] Got command RPT from MCS: ref #566909
2018-11-20 07:55:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 07:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566909  98 58442 28546646 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 07:56:00 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 07:56:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:56:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:56:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:56:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:56:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:56:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 07:56:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:56:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:56:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:56:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:56:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:56:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:56:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 07:56:19 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:56:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:56:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:56:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:56:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:56:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:56:19 [DEBUG   ] Finished updating board flags in 2.223 seconds
2018-11-20 07:56:20 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 07:56:40 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 07:57:00 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 07:57:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:57:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:57:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:57:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:57:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:57:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 07:57:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:57:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:57:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:57:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:57:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:57:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:57:19 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:57:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:57:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:57:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:57:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:57:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:57:19 [DEBUG   ] Finished updating board flags in 2.220 seconds
2018-11-20 07:57:20 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 07:57:40 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 07:58:00 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 07:58:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:58:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:58:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:58:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:58:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:58:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:58:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:58:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:58:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:58:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:58:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:58:19 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:58:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:58:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:58:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:58:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:58:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:58:19 [DEBUG   ] Finished updating board flags in 2.105 seconds
2018-11-20 07:58:21 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 07:58:41 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 07:59:01 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 07:59:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 07:59:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 07:59:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 07:59:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 07:59:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 07:59:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 07:59:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 07:59:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 07:59:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 07:59:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 07:59:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 07:59:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 07:59:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 07:59:19 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 07:59:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:59:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:59:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:59:19 [WARNING ] (13, 1, 'bot')
2018-11-20 07:59:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 07:59:19 [DEBUG   ] Finished updating board flags in 2.022 seconds
2018-11-20 07:59:21 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 07:59:41 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 08:00:01 [DEBUG   ] Finished updating ant stats in 1.006 seconds
2018-11-20 08:00:10 [DEBUG   ] Got command RPT from MCS: ref #566931
2018-11-20 08:00:10 [DEBUG   ] summary = ERROR
2018-11-20 08:00:10 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566931  13 58442 28810997 A  ERRORERROR'
2018-11-20 08:00:11 [DEBUG   ] Got command RPT from MCS: ref #566932
2018-11-20 08:00:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:00:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566932  98 58442 28811201 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:00:11 [DEBUG   ] Got command RPT from MCS: ref #566933
2018-11-20 08:00:11 [DEBUG   ] summary = ERROR
2018-11-20 08:00:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566933  13 58442 28811394 A  ERRORERROR'
2018-11-20 08:00:11 [DEBUG   ] Got command RPT from MCS: ref #566935
2018-11-20 08:00:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:00:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566935  98 58442 28811598 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:00:11 [DEBUG   ] Got command RPT from MCS: ref #566937
2018-11-20 08:00:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:00:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566937  12 58442 28811801 A  ERRORB¡ff'
2018-11-20 08:00:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:00:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:00:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:00:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:00:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:00:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 08:00:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:00:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:00:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:00:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:00:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:00:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:00:19 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:00:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:00:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:00:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:00:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:00:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:00:19 [DEBUG   ] Finished updating board flags in 2.003 seconds
2018-11-20 08:00:21 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 08:00:41 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 08:00:46 [DEBUG   ] Got command RPT from MCS: ref #566941
2018-11-20 08:00:46 [DEBUG   ] summary = ERROR
2018-11-20 08:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566941  13 58442 28846653 A  ERRORERROR'
2018-11-20 08:00:46 [DEBUG   ] Got command RPT from MCS: ref #566942
2018-11-20 08:00:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566942  98 58442 28846654 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:01:01 [DEBUG   ] Finished updating ant stats in 0.606 seconds
2018-11-20 08:01:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:01:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:01:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:01:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:01:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:01:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 08:01:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:01:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:01:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:01:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:01:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:01:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:01:19 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:01:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:01:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:01:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:01:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:01:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:01:19 [DEBUG   ] Finished updating board flags in 1.967 seconds
2018-11-20 08:01:21 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 08:01:41 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 08:02:01 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-20 08:02:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:02:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:02:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:02:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:02:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:02:18 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 08:02:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:02:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 08:02:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:02:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:02:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:02:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:02:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:02:19 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:02:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:02:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:02:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:02:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:02:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:02:19 [DEBUG   ] Finished updating board flags in 1.822 seconds
2018-11-20 08:02:22 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 08:02:42 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 08:03:02 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 08:03:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:03:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:03:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:03:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:03:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:03:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:03:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:03:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:03:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:03:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:03:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:03:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 08:03:19 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:03:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 08:03:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:03:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:03:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:03:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:03:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:03:19 [DEBUG   ] Finished updating board flags in 1.794 seconds
2018-11-20 08:03:22 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 08:03:42 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 08:04:02 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 08:04:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:04:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369407956000000
2018-11-20 08:04:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:04:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:04:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:04:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:04:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369407956000000
2018-11-20 08:04:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 08:04:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 08:04:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:04:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:04:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:04:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:04:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:04:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:04:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 08:04:19 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:04:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:04:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:04:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:04:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:04:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:04:19 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:04:19 [DEBUG   ] Finished updating board flags in 1.596 seconds
2018-11-20 08:04:22 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 08:04:42 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 08:05:02 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 08:05:11 [DEBUG   ] Got command RPT from MCS: ref #566966
2018-11-20 08:05:11 [DEBUG   ] summary = ERROR
2018-11-20 08:05:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566966  13 58442 29111073 A  ERRORERROR'
2018-11-20 08:05:11 [DEBUG   ] Got command RPT from MCS: ref #566967
2018-11-20 08:05:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:05:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566967  71 58442 29111277 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:05:11 [DEBUG   ] Got command RPT from MCS: ref #566968
2018-11-20 08:05:11 [DEBUG   ] summary = ERROR
2018-11-20 08:05:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566968  13 58442 29111481 A  ERRORERROR'
2018-11-20 08:05:11 [DEBUG   ] Got command RPT from MCS: ref #566969
2018-11-20 08:05:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:05:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566969  71 58442 29111685 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:05:11 [DEBUG   ] Got command RPT from MCS: ref #566970
2018-11-20 08:05:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:05:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566970  12 58442 29111877 A  ERRORB ff'
2018-11-20 08:05:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:05:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369419716000000
2018-11-20 08:05:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:05:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:05:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:05:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:05:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369419716000000
2018-11-20 08:05:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 08:05:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:05:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:05:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:05:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:05:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:05:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:05:19 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:05:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:05:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:05:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:05:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:05:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:05:19 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:05:19 [DEBUG   ] Finished updating board flags in 1.400 seconds
2018-11-20 08:05:22 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 08:05:43 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 08:05:46 [DEBUG   ] Got command RPT from MCS: ref #566971
2018-11-20 08:05:46 [DEBUG   ] summary = ERROR
2018-11-20 08:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566971  13 58442 29146649 A  ERRORERROR'
2018-11-20 08:05:46 [DEBUG   ] Got command RPT from MCS: ref #566972
2018-11-20 08:05:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566972  71 58442 29146650 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:06:03 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 08:06:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:06:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369431476000000
2018-11-20 08:06:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:06:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:06:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:06:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:06:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369431476000000
2018-11-20 08:06:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:06:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:06:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:06:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:06:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:06:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:06:19 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:06:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 08:06:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:06:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:06:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:06:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:06:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:06:19 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:06:19 [DEBUG   ] Finished updating board flags in 1.407 seconds
2018-11-20 08:06:23 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 08:06:43 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 08:07:03 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 08:07:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:07:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369443236000000
2018-11-20 08:07:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:07:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:07:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:07:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:07:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369443236000000
2018-11-20 08:07:18 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 08:07:18 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:07:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:07:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:07:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:07:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:07:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:07:19 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:07:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 08:07:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 08:07:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:07:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:07:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:07:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:07:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:07:19 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:07:19 [DEBUG   ] Finished updating board flags in 1.392 seconds
2018-11-20 08:07:23 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 08:07:43 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 08:08:03 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 08:08:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:08:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369454996000000
2018-11-20 08:08:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:08:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:08:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:08:18 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:08:18 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369454996000000
2018-11-20 08:08:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:08:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:08:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369455192000000
2018-11-20 08:08:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:08:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:08:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:08:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:08:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369455192000000
2018-11-20 08:08:19 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:08:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:08:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:08:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:08:19 [WARNING ] (13, 1, 'bot')
2018-11-20 08:08:19 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:08:19 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 08:08:19 [DEBUG   ] Finished updating board flags in 1.359 seconds
2018-11-20 08:08:23 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 08:08:43 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 08:09:03 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 08:09:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:09:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:09:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:09:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:09:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:09:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:09:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:09:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:09:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:09:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:09:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:09:20 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:09:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 08:09:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:09:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:09:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:09:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:09:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:09:20 [DEBUG   ] Finished updating board flags in 2.064 seconds
2018-11-20 08:09:23 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 08:09:43 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-20 08:10:03 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 08:10:11 [DEBUG   ] Got command RPT from MCS: ref #566994
2018-11-20 08:10:11 [DEBUG   ] summary = ERROR
2018-11-20 08:10:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566994  13 58442 29411161 A  ERRORERROR'
2018-11-20 08:10:11 [DEBUG   ] Got command RPT from MCS: ref #566995
2018-11-20 08:10:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:10:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566995  98 58442 29411365 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:10:11 [DEBUG   ] Got command RPT from MCS: ref #566996
2018-11-20 08:10:11 [DEBUG   ] summary = ERROR
2018-11-20 08:10:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566996  13 58442 29411557 A  ERRORERROR'
2018-11-20 08:10:11 [DEBUG   ] Got command RPT from MCS: ref #566998
2018-11-20 08:10:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:10:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566998  98 58442 29411761 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:10:11 [DEBUG   ] Got command RPT from MCS: ref #566999
2018-11-20 08:10:11 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:10:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   566999  12 58442 29411965 A  ERRORB£ff'
2018-11-20 08:10:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:10:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:10:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:10:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:10:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:10:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 08:10:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:10:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:10:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:10:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:10:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:10:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:10:20 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:10:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 08:10:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:10:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:10:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:10:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:10:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:10:20 [DEBUG   ] Finished updating board flags in 1.986 seconds
2018-11-20 08:10:23 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 08:10:43 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 08:10:46 [DEBUG   ] Got command RPT from MCS: ref #567008
2018-11-20 08:10:46 [DEBUG   ] summary = ERROR
2018-11-20 08:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567008  13 58442 29446653 A  ERRORERROR'
2018-11-20 08:10:46 [DEBUG   ] Got command RPT from MCS: ref #567009
2018-11-20 08:10:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567009  98 58442 29446654 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:11:03 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 08:11:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:11:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:11:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:11:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:11:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:11:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:11:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 08:11:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:11:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:11:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:11:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:11:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:11:20 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:11:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 08:11:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:11:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:11:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:11:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:11:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:11:20 [DEBUG   ] Finished updating board flags in 1.819 seconds
2018-11-20 08:11:23 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 08:11:44 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 08:12:04 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 08:12:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:12:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:12:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:12:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:12:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:12:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:12:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:12:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:12:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:12:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:12:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:12:20 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:12:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:12:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:12:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:12:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:12:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:12:20 [DEBUG   ] Finished updating board flags in 1.808 seconds
2018-11-20 08:12:24 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 08:12:44 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 08:13:04 [DEBUG   ] Finished updating ant stats in 1.005 seconds
2018-11-20 08:13:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:13:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:13:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:13:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:13:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:13:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:13:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:13:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:13:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:13:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:13:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:13:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 1
2018-11-20 08:13:20 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:13:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 08:13:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:13:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:13:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:13:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:13:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:13:20 [DEBUG   ] Finished updating board flags in 1.785 seconds
2018-11-20 08:13:24 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 08:13:44 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 08:14:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 08:14:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:14:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369525752000000
2018-11-20 08:14:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:14:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:14:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:14:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:14:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369525752000000
2018-11-20 08:14:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:14:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:14:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:14:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:14:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:14:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:14:20 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:14:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 08:14:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:14:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:14:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:14:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:14:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:14:20 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:14:20 [DEBUG   ] Finished updating board flags in 1.609 seconds
2018-11-20 08:14:24 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 08:14:44 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 08:15:04 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 08:15:11 [DEBUG   ] Got command RPT from MCS: ref #567029
2018-11-20 08:15:11 [DEBUG   ] summary = ERROR
2018-11-20 08:15:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567029  13 58442 29711241 A  ERRORERROR'
2018-11-20 08:15:11 [DEBUG   ] Got command RPT from MCS: ref #567030
2018-11-20 08:15:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:15:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567030  71 58442 29711445 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:15:11 [DEBUG   ] Got command RPT from MCS: ref #567031
2018-11-20 08:15:11 [DEBUG   ] summary = ERROR
2018-11-20 08:15:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567031  13 58442 29711637 A  ERRORERROR'
2018-11-20 08:15:11 [DEBUG   ] Got command RPT from MCS: ref #567032
2018-11-20 08:15:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:15:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567032  71 58442 29711841 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:15:12 [DEBUG   ] Got command RPT from MCS: ref #567033
2018-11-20 08:15:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:15:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567033  12 58442 29712045 A  ERRORB¢ff'
2018-11-20 08:15:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:15:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369537512000000
2018-11-20 08:15:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:15:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:15:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:15:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:15:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369537512000000
2018-11-20 08:15:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:15:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:15:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:15:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:15:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:15:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:15:20 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:15:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:15:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:15:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:15:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:15:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:15:20 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:15:20 [DEBUG   ] Finished updating board flags in 1.586 seconds
2018-11-20 08:15:24 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 08:15:44 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 08:15:46 [DEBUG   ] Got command RPT from MCS: ref #567034
2018-11-20 08:15:46 [DEBUG   ] summary = ERROR
2018-11-20 08:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567034  13 58442 29746645 A  ERRORERROR'
2018-11-20 08:15:46 [DEBUG   ] Got command RPT from MCS: ref #567035
2018-11-20 08:15:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567035  71 58442 29746646 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:16:04 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 08:16:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:16:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369549272000000
2018-11-20 08:16:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:16:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:16:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:16:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:16:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369549272000000
2018-11-20 08:16:19 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 08:16:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:16:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:16:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:16:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:16:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:16:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:16:20 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:16:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 08:16:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:16:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:16:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:16:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:16:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:16:20 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:16:20 [DEBUG   ] Finished updating board flags in 1.423 seconds
2018-11-20 08:16:24 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 08:16:45 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 08:17:05 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 08:17:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:17:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369561032000000
2018-11-20 08:17:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:17:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:17:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:17:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:17:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369561032000000
2018-11-20 08:17:19 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:17:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:17:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:17:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:17:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:17:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:17:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 08:17:20 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:17:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:17:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:17:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:17:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:17:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:17:20 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:17:20 [DEBUG   ] Finished updating board flags in 1.383 seconds
2018-11-20 08:17:25 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 08:17:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 08:18:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 08:18:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:18:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369572792000000
2018-11-20 08:18:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:18:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:18:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:18:19 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:18:19 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369572792000000
2018-11-20 08:18:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:18:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:18:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369572988000000
2018-11-20 08:18:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:18:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:18:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:18:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:18:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369572988000000
2018-11-20 08:18:20 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:18:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:18:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:18:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:18:20 [WARNING ] (13, 1, 'bot')
2018-11-20 08:18:20 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:18:20 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 08:18:20 [DEBUG   ] Finished updating board flags in 1.369 seconds
2018-11-20 08:18:25 [DEBUG   ] Finished updating ant stats in 1.005 seconds
2018-11-20 08:18:45 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 08:19:05 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 08:19:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:19:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:19:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:19:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:19:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:19:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 08:19:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:19:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:19:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:19:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:19:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:19:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:19:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 08:19:21 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:19:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:19:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:19:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:19:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:19:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:19:21 [DEBUG   ] Finished updating board flags in 2.056 seconds
2018-11-20 08:19:25 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 08:19:45 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 08:20:05 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 08:20:11 [DEBUG   ] Got command RPT from MCS: ref #567061
2018-11-20 08:20:11 [DEBUG   ] summary = ERROR
2018-11-20 08:20:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567061  13 58442 30011325 A  ERRORERROR'
2018-11-20 08:20:11 [DEBUG   ] Got command RPT from MCS: ref #567062
2018-11-20 08:20:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:20:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567062  98 58442 30011517 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:20:11 [DEBUG   ] Got command RPT from MCS: ref #567063
2018-11-20 08:20:11 [DEBUG   ] summary = ERROR
2018-11-20 08:20:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567063  13 58442 30011721 A  ERRORERROR'
2018-11-20 08:20:11 [DEBUG   ] Got command RPT from MCS: ref #567065
2018-11-20 08:20:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:20:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567065  98 58442 30011940 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:20:12 [DEBUG   ] Got command RPT from MCS: ref #567067
2018-11-20 08:20:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:20:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567067  12 58442 30012117 A  ERRORB¡ff'
2018-11-20 08:20:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:20:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:20:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:20:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:20:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:20:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:20:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:20:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:20:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:20:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:20:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:20:21 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:20:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:20:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:20:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:20:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:20:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:20:21 [DEBUG   ] Finished updating board flags in 1.988 seconds
2018-11-20 08:20:25 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 08:20:46 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 08:20:46 [DEBUG   ] Got command RPT from MCS: ref #567071
2018-11-20 08:20:46 [DEBUG   ] summary = ERROR
2018-11-20 08:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567071  13 58442 30046649 A  ERRORERROR'
2018-11-20 08:20:46 [DEBUG   ] Got command RPT from MCS: ref #567072
2018-11-20 08:20:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567072  98 58442 30046650 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:21:05 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-20 08:21:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:21:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:21:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:21:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:21:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:21:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 08:21:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:21:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:21:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:21:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:21:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:21:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:21:21 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:21:21 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 08:21:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:21:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:21:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:21:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:21:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:21:21 [DEBUG   ] Finished updating board flags in 1.806 seconds
2018-11-20 08:21:26 [DEBUG   ] Finished updating ant stats in 1.392 seconds
2018-11-20 08:21:46 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 08:22:06 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 08:22:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:22:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:22:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:22:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:22:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:22:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:22:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:22:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:22:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:22:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:22:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:22:21 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:22:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:22:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:22:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:22:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:22:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:22:21 [DEBUG   ] Finished updating board flags in 1.803 seconds
2018-11-20 08:22:26 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 08:22:46 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 08:23:06 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 08:23:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:23:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:23:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:23:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:23:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:23:20 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 08:23:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 08:23:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:23:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:23:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:23:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:23:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:23:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:23:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 08:23:21 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:23:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:23:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:23:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:23:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:23:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:23:21 [DEBUG   ] Finished updating board flags in 1.789 seconds
2018-11-20 08:23:26 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 08:23:46 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 08:24:06 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 08:24:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:24:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369643548000000
2018-11-20 08:24:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:24:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:24:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:24:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:24:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369643548000000
2018-11-20 08:24:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:24:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:24:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:24:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:24:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:24:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:24:21 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:24:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:24:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:24:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:24:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:24:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:24:21 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:24:21 [DEBUG   ] Finished updating board flags in 1.621 seconds
2018-11-20 08:24:26 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 08:24:46 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 08:25:06 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 08:25:11 [DEBUG   ] Got command RPT from MCS: ref #567092
2018-11-20 08:25:11 [DEBUG   ] summary = ERROR
2018-11-20 08:25:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567092  13 58442 30311401 A  ERRORERROR'
2018-11-20 08:25:11 [DEBUG   ] Got command RPT from MCS: ref #567093
2018-11-20 08:25:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:25:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567093  71 58442 30311605 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:25:11 [DEBUG   ] Got command RPT from MCS: ref #567094
2018-11-20 08:25:11 [DEBUG   ] summary = ERROR
2018-11-20 08:25:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567094  13 58442 30311797 A  ERRORERROR'
2018-11-20 08:25:12 [DEBUG   ] Got command RPT from MCS: ref #567095
2018-11-20 08:25:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:25:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567095  71 58442 30312001 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:25:12 [DEBUG   ] Got command RPT from MCS: ref #567096
2018-11-20 08:25:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:25:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567096  12 58442 30312205 A  ERRORB ff'
2018-11-20 08:25:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:25:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369655308000000
2018-11-20 08:25:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:25:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:25:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:25:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:25:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369655308000000
2018-11-20 08:25:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 08:25:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:25:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:25:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:25:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:25:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:25:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:25:21 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:25:21 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 08:25:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:25:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:25:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:25:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:25:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:25:21 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:25:21 [DEBUG   ] Finished updating board flags in 1.583 seconds
2018-11-20 08:25:26 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 08:25:46 [DEBUG   ] Got command RPT from MCS: ref #567101
2018-11-20 08:25:46 [DEBUG   ] summary = ERROR
2018-11-20 08:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567101  13 58442 30346649 A  ERRORERROR'
2018-11-20 08:25:46 [DEBUG   ] Got command RPT from MCS: ref #567102
2018-11-20 08:25:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567102  71 58442 30346650 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:25:47 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 08:26:07 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 08:26:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:26:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369667068000000
2018-11-20 08:26:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:26:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:26:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:26:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:26:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369667068000000
2018-11-20 08:26:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:26:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:26:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:26:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:26:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:26:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:26:21 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:26:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 08:26:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:26:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:26:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:26:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:26:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:26:21 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:26:21 [DEBUG   ] Finished updating board flags in 1.419 seconds
2018-11-20 08:26:27 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 08:26:47 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 08:27:07 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 08:27:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:27:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369678828000000
2018-11-20 08:27:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:27:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:27:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:27:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:27:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369678828000000
2018-11-20 08:27:20 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:27:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:27:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:27:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:27:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:27:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:27:21 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:27:21 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 08:27:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:27:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:27:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:27:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:27:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:27:21 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:27:21 [DEBUG   ] Finished updating board flags in 1.390 seconds
2018-11-20 08:27:27 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 08:27:47 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 08:28:07 [DEBUG   ] Finished updating ant stats in 1.005 seconds
2018-11-20 08:28:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:28:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369690588000000
2018-11-20 08:28:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:28:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:28:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:28:20 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:28:20 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369690588000000
2018-11-20 08:28:20 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 08:28:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:28:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:28:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369690784000000
2018-11-20 08:28:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:28:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:28:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:28:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:28:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369690784000000
2018-11-20 08:28:21 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:28:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:28:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:28:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:28:21 [WARNING ] (13, 1, 'bot')
2018-11-20 08:28:21 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:28:21 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 08:28:21 [DEBUG   ] Finished updating board flags in 1.367 seconds
2018-11-20 08:28:27 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 08:28:47 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 08:29:07 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 08:29:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:29:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:29:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:29:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:29:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:29:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:29:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:29:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:29:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:29:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:29:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:29:22 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:29:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 08:29:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:29:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:29:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:29:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:29:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:29:22 [DEBUG   ] Finished updating board flags in 2.049 seconds
2018-11-20 08:29:27 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 08:29:47 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 08:30:07 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 08:30:11 [DEBUG   ] Got command RPT from MCS: ref #567124
2018-11-20 08:30:11 [DEBUG   ] summary = ERROR
2018-11-20 08:30:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567124  13 58442 30611477 A  ERRORERROR'
2018-11-20 08:30:11 [DEBUG   ] Got command RPT from MCS: ref #567125
2018-11-20 08:30:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:30:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567125  98 58442 30611681 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:30:11 [DEBUG   ] Got command RPT from MCS: ref #567126
2018-11-20 08:30:11 [DEBUG   ] summary = ERROR
2018-11-20 08:30:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567126  13 58442 30611885 A  ERRORERROR'
2018-11-20 08:30:12 [DEBUG   ] Got command RPT from MCS: ref #567128
2018-11-20 08:30:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:30:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567128  98 58442 30612077 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:30:12 [DEBUG   ] Got command RPT from MCS: ref #567129
2018-11-20 08:30:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:30:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567129  12 58442 30612281 A  ERRORB ff'
2018-11-20 08:30:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:30:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:30:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:30:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:30:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:30:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:30:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:30:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:30:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:30:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:30:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:30:22 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:30:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:30:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:30:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:30:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:30:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:30:22 [DEBUG   ] Finished updating board flags in 1.991 seconds
2018-11-20 08:30:27 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 08:30:46 [DEBUG   ] Got command RPT from MCS: ref #567134
2018-11-20 08:30:46 [DEBUG   ] summary = ERROR
2018-11-20 08:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567134  13 58442 30646649 A  ERRORERROR'
2018-11-20 08:30:46 [DEBUG   ] Got command RPT from MCS: ref #567135
2018-11-20 08:30:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567135  98 58442 30646650 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:30:47 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 08:31:07 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 08:31:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:31:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:31:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:31:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:31:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:31:21 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 08:31:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:31:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:31:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:31:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:31:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:31:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:31:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 08:31:22 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:31:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:31:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:31:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:31:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:31:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:31:22 [DEBUG   ] Finished updating board flags in 1.845 seconds
2018-11-20 08:31:27 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 08:31:47 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 08:32:08 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 08:32:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:32:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:32:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:32:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:32:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:32:21 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 08:32:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:32:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:32:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:32:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:32:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:32:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:32:22 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:32:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:32:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:32:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:32:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:32:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:32:22 [DEBUG   ] Finished updating board flags in 1.763 seconds
2018-11-20 08:32:28 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 08:32:48 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 08:33:08 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 08:33:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:33:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369749584000000
2018-11-20 08:33:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:33:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:33:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:33:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:33:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369749584000000
2018-11-20 08:33:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 08:33:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 08:33:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:33:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:33:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:33:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:33:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:33:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:33:22 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:33:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:33:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:33:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:33:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:33:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:33:22 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:33:22 [DEBUG   ] Finished updating board flags in 1.613 seconds
2018-11-20 08:33:28 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 08:33:48 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 08:34:08 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 08:34:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:34:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369761344000000
2018-11-20 08:34:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:34:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:34:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:34:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:34:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369761344000000
2018-11-20 08:34:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:34:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:34:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:34:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:34:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:34:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:34:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 08:34:22 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:34:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 08:34:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:34:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:34:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:34:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:34:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:34:22 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:34:22 [DEBUG   ] Finished updating board flags in 1.590 seconds
2018-11-20 08:34:28 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 08:34:48 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 08:35:08 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 08:35:11 [DEBUG   ] Got command RPT from MCS: ref #567159
2018-11-20 08:35:11 [DEBUG   ] summary = ERROR
2018-11-20 08:35:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567159  13 58442 30911557 A  ERRORERROR'
2018-11-20 08:35:11 [DEBUG   ] Got command RPT from MCS: ref #567160
2018-11-20 08:35:11 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:35:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567160  71 58442 30911761 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:35:11 [DEBUG   ] Got command RPT from MCS: ref #567161
2018-11-20 08:35:11 [DEBUG   ] summary = ERROR
2018-11-20 08:35:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567161  13 58442 30911965 A  ERRORERROR'
2018-11-20 08:35:12 [DEBUG   ] Got command RPT from MCS: ref #567162
2018-11-20 08:35:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:35:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567162  71 58442 30912157 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:35:12 [DEBUG   ] Got command RPT from MCS: ref #567163
2018-11-20 08:35:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:35:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567163  12 58442 30912361 A  ERRORB ff'
2018-11-20 08:35:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:35:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369773104000000
2018-11-20 08:35:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:35:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:35:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:35:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:35:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369773104000000
2018-11-20 08:35:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:35:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:35:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:35:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:35:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:35:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:35:22 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:35:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:35:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:35:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:35:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:35:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:35:22 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:35:22 [DEBUG   ] Finished updating board flags in 1.407 seconds
2018-11-20 08:35:28 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-20 08:35:46 [DEBUG   ] Got command RPT from MCS: ref #567164
2018-11-20 08:35:46 [DEBUG   ] summary = ERROR
2018-11-20 08:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567164  13 58442 30946658 A  ERRORERROR'
2018-11-20 08:35:46 [DEBUG   ] Got command RPT from MCS: ref #567165
2018-11-20 08:35:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567165  71 58442 30946658 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:35:48 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 08:36:08 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 08:36:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:36:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369784864000000
2018-11-20 08:36:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:36:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:36:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:36:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:36:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369784864000000
2018-11-20 08:36:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:36:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:36:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:36:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:36:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:36:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:36:22 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:36:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:36:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:36:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:36:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:36:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:36:22 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:36:22 [DEBUG   ] Finished updating board flags in 1.412 seconds
2018-11-20 08:36:29 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 08:36:49 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 08:37:09 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 08:37:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:37:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369796624000000
2018-11-20 08:37:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:37:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:37:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:37:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:37:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369796624000000
2018-11-20 08:37:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 08:37:21 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 08:37:21 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:37:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:37:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:37:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:37:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:37:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:37:22 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:37:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:37:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:37:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:37:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:37:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:37:22 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:37:22 [DEBUG   ] Finished updating board flags in 1.383 seconds
2018-11-20 08:37:29 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 08:37:49 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 08:38:09 [DEBUG   ] Finished updating ant stats in 1.005 seconds
2018-11-20 08:38:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:38:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369808384000000
2018-11-20 08:38:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:38:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:38:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:38:21 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:38:21 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369808384000000
2018-11-20 08:38:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:38:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:38:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369808580000000
2018-11-20 08:38:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:38:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:38:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:38:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:38:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369808580000000
2018-11-20 08:38:22 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:38:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 08:38:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:38:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:38:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:38:22 [WARNING ] (13, 1, 'bot')
2018-11-20 08:38:22 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:38:22 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 08:38:22 [DEBUG   ] Finished updating board flags in 1.356 seconds
2018-11-20 08:38:29 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 08:38:49 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 08:39:09 [DEBUG   ] Finished updating ant stats in 0.805 seconds
2018-11-20 08:39:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:39:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:39:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:39:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:39:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:39:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 08:39:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:39:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 08:39:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:39:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:39:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:39:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:39:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:39:23 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:39:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:39:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:39:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:39:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:39:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:39:23 [DEBUG   ] Finished updating board flags in 2.053 seconds
2018-11-20 08:39:29 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 08:39:49 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 08:40:09 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 08:40:11 [DEBUG   ] Got command RPT from MCS: ref #567187
2018-11-20 08:40:11 [DEBUG   ] summary = ERROR
2018-11-20 08:40:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567187  13 58442 31211634 A  ERRORERROR'
2018-11-20 08:40:11 [DEBUG   ] Got command RPT from MCS: ref #567188
2018-11-20 08:40:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:40:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567188  98 58442 31211837 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:40:12 [DEBUG   ] Got command RPT from MCS: ref #567189
2018-11-20 08:40:12 [DEBUG   ] summary = ERROR
2018-11-20 08:40:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567189  13 58442 31212041 A  ERRORERROR'
2018-11-20 08:40:12 [DEBUG   ] Got command RPT from MCS: ref #567191
2018-11-20 08:40:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:40:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567191  98 58442 31212245 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:40:12 [DEBUG   ] Got command RPT from MCS: ref #567192
2018-11-20 08:40:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:40:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567192  12 58442 31212438 A  ERRORB£ff'
2018-11-20 08:40:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:40:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:40:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:40:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:40:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:40:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 08:40:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:40:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:40:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:40:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:40:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:40:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:40:23 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:40:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:40:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:40:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:40:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:40:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:40:23 [DEBUG   ] Finished updating board flags in 2.007 seconds
2018-11-20 08:40:29 [DEBUG   ] Finished updating ant stats in 0.605 seconds
2018-11-20 08:40:46 [DEBUG   ] Got command RPT from MCS: ref #567201
2018-11-20 08:40:46 [DEBUG   ] summary = ERROR
2018-11-20 08:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567201  13 58442 31246649 A  ERRORERROR'
2018-11-20 08:40:46 [DEBUG   ] Got command RPT from MCS: ref #567202
2018-11-20 08:40:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567202  98 58442 31246650 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:40:49 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 08:41:10 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 08:41:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:41:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:41:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:41:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:41:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:41:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:41:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:41:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:41:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:41:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:41:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:41:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 08:41:23 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:41:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 08:41:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:41:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:41:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:41:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:41:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:41:23 [DEBUG   ] Finished updating board flags in 1.988 seconds
2018-11-20 08:41:30 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 08:41:50 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 08:42:10 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 08:42:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:42:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:42:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:42:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:42:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:42:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:42:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 08:42:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:42:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:42:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:42:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:42:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:42:23 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:42:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:42:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:42:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:42:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:42:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:42:23 [DEBUG   ] Finished updating board flags in 1.816 seconds
2018-11-20 08:42:30 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 08:42:50 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 08:43:10 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 08:43:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:43:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:43:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:43:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:43:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:43:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 08:43:22 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 08:43:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:43:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:43:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:43:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:43:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:43:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:43:23 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:43:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:43:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:43:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:43:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:43:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:43:23 [DEBUG   ] Finished updating board flags in 1.858 seconds
2018-11-20 08:43:30 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 08:43:50 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 08:44:10 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 08:44:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:44:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:44:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:44:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:44:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:44:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:44:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:44:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:44:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:44:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:44:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:44:23 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:44:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:44:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:44:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:44:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:44:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:44:23 [DEBUG   ] Finished updating board flags in 1.824 seconds
2018-11-20 08:44:30 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 08:44:50 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 08:45:10 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 08:45:11 [DEBUG   ] Got command RPT from MCS: ref #567222
2018-11-20 08:45:11 [DEBUG   ] summary = ERROR
2018-11-20 08:45:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567222  13 58442 31511721 A  ERRORERROR'
2018-11-20 08:45:11 [DEBUG   ] Got command RPT from MCS: ref #567223
2018-11-20 08:45:11 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:45:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567223  98 58442 31511913 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:45:12 [DEBUG   ] Got command RPT from MCS: ref #567224
2018-11-20 08:45:12 [DEBUG   ] summary = ERROR
2018-11-20 08:45:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567224  13 58442 31512117 A  ERRORERROR'
2018-11-20 08:45:12 [DEBUG   ] Got command RPT from MCS: ref #567225
2018-11-20 08:45:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:45:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567225  98 58442 31512321 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:45:12 [DEBUG   ] Got command RPT from MCS: ref #567226
2018-11-20 08:45:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:45:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567226  12 58442 31512525 A  ERRORB¢ff'
2018-11-20 08:45:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:45:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:45:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:45:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:45:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:45:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:45:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:45:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:45:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:45:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:45:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:45:23 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:45:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:45:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:45:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:45:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:45:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:45:23 [DEBUG   ] Finished updating board flags in 1.712 seconds
2018-11-20 08:45:30 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 08:45:46 [DEBUG   ] Got command RPT from MCS: ref #567227
2018-11-20 08:45:46 [DEBUG   ] summary = ERROR
2018-11-20 08:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567227  13 58442 31546649 A  ERRORERROR'
2018-11-20 08:45:46 [DEBUG   ] Got command RPT from MCS: ref #567228
2018-11-20 08:45:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567228  98 58442 31546650 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:45:50 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 08:46:10 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 08:46:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:46:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369902660000000
2018-11-20 08:46:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:46:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:46:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:46:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:46:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369902660000000
2018-11-20 08:46:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:46:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:46:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:46:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:46:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:46:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:46:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 08:46:23 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:46:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:46:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:46:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:46:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:46:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:46:23 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:46:23 [DEBUG   ] Finished updating board flags in 1.615 seconds
2018-11-20 08:46:31 [DEBUG   ] Finished updating ant stats in 1.401 seconds
2018-11-20 08:46:51 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 08:47:11 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 08:47:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:47:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369914420000000
2018-11-20 08:47:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:47:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:47:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:47:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:47:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369914420000000
2018-11-20 08:47:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:47:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:47:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:47:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:47:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:47:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:47:23 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:47:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 08:47:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:47:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:47:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:47:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:47:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:47:23 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:47:23 [DEBUG   ] Finished updating board flags in 1.592 seconds
2018-11-20 08:47:31 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 08:47:51 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 08:48:11 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 08:48:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:48:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369926180000000
2018-11-20 08:48:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:48:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:48:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:48:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:48:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369926180000000
2018-11-20 08:48:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 08:48:22 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 08:48:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:48:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:48:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:48:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:48:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:48:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:48:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 08:48:23 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:48:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:48:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:48:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:48:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:48:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:48:23 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:48:23 [DEBUG   ] Finished updating board flags in 1.425 seconds
2018-11-20 08:48:31 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 08:48:51 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 08:49:11 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 08:49:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:49:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369937940000000
2018-11-20 08:49:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:49:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:49:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:49:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:49:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369937940000000
2018-11-20 08:49:22 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:49:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:49:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:49:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:49:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:49:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:49:23 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:49:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 08:49:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:49:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:49:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:49:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:49:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:49:23 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:49:23 [DEBUG   ] Finished updating board flags in 1.370 seconds
2018-11-20 08:49:31 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 08:49:51 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 08:50:11 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 08:50:11 [DEBUG   ] Got command RPT from MCS: ref #567254
2018-11-20 08:50:11 [DEBUG   ] summary = ERROR
2018-11-20 08:50:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567254  13 58442 31811797 A  ERRORERROR'
2018-11-20 08:50:12 [DEBUG   ] Got command RPT from MCS: ref #567255
2018-11-20 08:50:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:50:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567255  71 58442 31812001 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:50:12 [DEBUG   ] Got command RPT from MCS: ref #567256
2018-11-20 08:50:12 [DEBUG   ] summary = ERROR
2018-11-20 08:50:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567256  13 58442 31812197 A  ERRORERROR'
2018-11-20 08:50:12 [DEBUG   ] Got command RPT from MCS: ref #567258
2018-11-20 08:50:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 08:50:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567258  71 58442 31812401 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 08:50:12 [DEBUG   ] Got command RPT from MCS: ref #567259
2018-11-20 08:50:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:50:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567259  12 58442 31812605 A  ERRORB¡ff'
2018-11-20 08:50:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:50:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302369949700000000
2018-11-20 08:50:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:50:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:50:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:50:22 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:50:22 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302369949700000000
2018-11-20 08:50:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:50:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:50:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302369949896000000
2018-11-20 08:50:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:50:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:50:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:50:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:50:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302369949896000000
2018-11-20 08:50:23 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:50:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 08:50:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:50:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:50:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:50:23 [WARNING ] (13, 1, 'bot')
2018-11-20 08:50:23 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:50:23 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 08:50:23 [DEBUG   ] Finished updating board flags in 1.396 seconds
2018-11-20 08:50:31 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 08:50:46 [DEBUG   ] Got command RPT from MCS: ref #567264
2018-11-20 08:50:46 [DEBUG   ] summary = ERROR
2018-11-20 08:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567264  13 58442 31846649 A  ERRORERROR'
2018-11-20 08:50:46 [DEBUG   ] Got command RPT from MCS: ref #567265
2018-11-20 08:50:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 08:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567265  79 58442 31846650 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 08:50:51 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 08:51:11 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 08:51:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:51:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:51:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:51:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:51:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:51:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:51:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:51:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:51:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:51:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:51:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:51:24 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:51:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:51:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:51:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:51:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:51:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:51:24 [DEBUG   ] Finished updating board flags in 2.030 seconds
2018-11-20 08:51:31 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 08:51:51 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 08:52:11 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 08:52:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:52:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:52:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:52:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:52:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:52:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 08:52:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:52:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:52:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:52:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:52:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:52:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:52:24 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:52:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:52:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:52:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:52:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:52:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:52:24 [DEBUG   ] Finished updating board flags in 2.018 seconds
2018-11-20 08:52:32 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 08:52:52 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 08:53:12 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 08:53:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:53:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:53:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:53:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:53:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:53:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:53:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:53:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:53:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:53:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:53:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:53:24 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:53:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:53:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:53:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:53:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:53:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:53:24 [DEBUG   ] Finished updating board flags in 1.968 seconds
2018-11-20 08:53:32 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 08:53:52 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 08:54:12 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 08:54:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:54:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:54:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:54:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:54:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:54:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:54:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:54:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:54:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:54:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:54:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:54:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 08:54:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 08:54:24 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:54:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:54:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:54:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:54:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:54:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:54:24 [DEBUG   ] Finished updating board flags in 1.805 seconds
2018-11-20 08:54:32 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 08:54:52 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 08:55:11 [DEBUG   ] Got command RPT from MCS: ref #567285
2018-11-20 08:55:11 [DEBUG   ] summary = ERROR
2018-11-20 08:55:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567285  13 58442 32111881 A  ERRORERROR'
2018-11-20 08:55:12 [DEBUG   ] Got command RPT from MCS: ref #567286
2018-11-20 08:55:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:55:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567286  98 58442 32112073 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:55:12 [DEBUG   ] Got command RPT from MCS: ref #567287
2018-11-20 08:55:12 [DEBUG   ] summary = ERROR
2018-11-20 08:55:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567287  13 58442 32112277 A  ERRORERROR'
2018-11-20 08:55:12 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 08:55:12 [DEBUG   ] Got command RPT from MCS: ref #567288
2018-11-20 08:55:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:55:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567288  98 58442 32112481 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:55:12 [DEBUG   ] Got command RPT from MCS: ref #567289
2018-11-20 08:55:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 08:55:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567289  12 58442 32112685 A  ERRORB¡ff'
2018-11-20 08:55:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:55:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:55:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:55:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:55:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:55:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 08:55:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:55:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:55:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:55:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:55:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:55:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:55:24 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:55:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:55:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:55:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:55:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:55:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:55:24 [DEBUG   ] Finished updating board flags in 1.938 seconds
2018-11-20 08:55:32 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 08:55:46 [DEBUG   ] Got command RPT from MCS: ref #567294
2018-11-20 08:55:46 [DEBUG   ] summary = ERROR
2018-11-20 08:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567294  13 58442 32146653 A  ERRORERROR'
2018-11-20 08:55:46 [DEBUG   ] Got command RPT from MCS: ref #567295
2018-11-20 08:55:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 08:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567295  98 58442 32146654 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 08:55:52 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 08:56:12 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 08:56:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:56:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:56:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:56:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:56:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:56:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 08:56:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 08:56:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:56:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:56:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:56:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:56:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:56:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:56:24 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:56:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:56:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:56:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:56:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:56:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:56:24 [DEBUG   ] Finished updating board flags in 1.651 seconds
2018-11-20 08:56:32 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 08:56:52 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 08:57:13 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-20 08:57:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:57:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370032216000000
2018-11-20 08:57:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:57:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:57:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:57:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:57:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370032216000000
2018-11-20 08:57:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 08:57:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:57:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:57:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:57:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:57:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:57:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:57:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 08:57:24 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:57:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:57:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:57:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:57:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:57:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:57:24 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:57:24 [DEBUG   ] Finished updating board flags in 1.626 seconds
2018-11-20 08:57:33 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 08:57:53 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 08:58:13 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 08:58:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:58:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370043976000000
2018-11-20 08:58:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:58:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:58:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:58:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:58:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370043976000000
2018-11-20 08:58:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-20 08:58:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:58:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:58:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:58:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:58:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:58:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:58:24 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 08:58:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:58:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:58:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:58:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:58:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:58:24 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:58:24 [DEBUG   ] Finished updating board flags in 1.579 seconds
2018-11-20 08:58:33 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 08:58:53 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 08:59:13 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 08:59:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 08:59:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370055736000000
2018-11-20 08:59:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 08:59:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 08:59:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 08:59:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 08:59:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370055736000000
2018-11-20 08:59:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 08:59:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 08:59:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 08:59:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 08:59:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 08:59:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 08:59:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 08:59:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 08:59:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 08:59:24 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 08:59:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 08:59:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:59:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:59:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:59:24 [WARNING ] (13, 1, 'bot')
2018-11-20 08:59:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 08:59:24 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 08:59:24 [DEBUG   ] Finished updating board flags in 1.443 seconds
2018-11-20 08:59:33 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 08:59:53 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 09:00:11 [DEBUG   ] Got command RPT from MCS: ref #567317
2018-11-20 09:00:11 [DEBUG   ] summary = ERROR
2018-11-20 09:00:11 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567317  13 58442 32411957 A  ERRORERROR'
2018-11-20 09:00:12 [DEBUG   ] Got command RPT from MCS: ref #567318
2018-11-20 09:00:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:00:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567318  71 58442 32412161 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:00:12 [DEBUG   ] Got command RPT from MCS: ref #567319
2018-11-20 09:00:12 [DEBUG   ] summary = ERROR
2018-11-20 09:00:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567319  13 58442 32412365 A  ERRORERROR'
2018-11-20 09:00:12 [DEBUG   ] Got command RPT from MCS: ref #567321
2018-11-20 09:00:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:00:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567321  71 58442 32412558 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:00:12 [DEBUG   ] Got command RPT from MCS: ref #567322
2018-11-20 09:00:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:00:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567322  12 58442 32412761 A  ERRORB¢ff'
2018-11-20 09:00:13 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:00:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:00:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370067496000000
2018-11-20 09:00:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:00:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:00:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:00:23 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:00:23 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370067496000000
2018-11-20 09:00:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 09:00:23 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 09:00:23 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:00:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:00:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:00:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:00:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:00:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:00:24 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:00:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 09:00:24 [WARNING ] (13, 1, 'bot')
2018-11-20 09:00:24 [WARNING ] (13, 1, 'bot')
2018-11-20 09:00:24 [WARNING ] (13, 1, 'bot')
2018-11-20 09:00:24 [WARNING ] (13, 1, 'bot')
2018-11-20 09:00:24 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:00:24 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:00:24 [DEBUG   ] Finished updating board flags in 1.373 seconds
2018-11-20 09:00:33 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 09:00:46 [DEBUG   ] Got command RPT from MCS: ref #567327
2018-11-20 09:00:46 [DEBUG   ] summary = ERROR
2018-11-20 09:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567327  13 58442 32446662 A  ERRORERROR'
2018-11-20 09:00:46 [DEBUG   ] Got command RPT from MCS: ref #567328
2018-11-20 09:00:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567328  71 58442 32446662 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:00:53 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 09:01:13 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 09:01:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:01:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370079256000000
2018-11-20 09:01:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:01:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:01:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:01:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:01:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:01:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-20 09:01:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:01:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:01:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:01:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:01:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:01:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:01:25 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 09:01:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:01:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 09:01:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:01:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:01:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:01:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:01:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:01:25 [CRITICAL] DRX time not updating correctly on beam(s) #4
2018-11-20 09:01:25 [DEBUG   ] Finished updating board flags in 2.209 seconds
2018-11-20 09:01:33 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 09:01:53 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 09:02:14 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 09:02:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:02:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:02:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:02:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:02:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:02:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:02:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 09:02:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:02:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:02:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:02:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:02:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:02:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:02:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:02:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:02:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:02:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:02:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:02:25 [DEBUG   ] Finished updating board flags in 2.192 seconds
2018-11-20 09:02:34 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 09:02:54 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 09:03:14 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 09:03:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:03:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:03:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:03:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:03:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:03:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:03:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:03:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:03:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:03:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:03:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:03:25 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:03:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 09:03:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:03:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:03:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:03:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:03:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:03:25 [DEBUG   ] Finished updating board flags in 2.012 seconds
2018-11-20 09:03:34 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:03:54 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:04:14 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 09:04:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:04:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:04:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:04:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:04:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:04:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:04:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:04:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:04:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:04:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:04:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:04:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:04:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:04:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:04:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:04:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:04:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:04:25 [DEBUG   ] Finished updating board flags in 1.993 seconds
2018-11-20 09:04:34 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 09:04:54 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:05:12 [DEBUG   ] Got command RPT from MCS: ref #567352
2018-11-20 09:05:12 [DEBUG   ] summary = ERROR
2018-11-20 09:05:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567352  13 58442 32712041 A  ERRORERROR'
2018-11-20 09:05:12 [DEBUG   ] Got command RPT from MCS: ref #567353
2018-11-20 09:05:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:05:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567353  98 58442 32712245 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:05:12 [DEBUG   ] Got command RPT from MCS: ref #567354
2018-11-20 09:05:12 [DEBUG   ] summary = ERROR
2018-11-20 09:05:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567354  13 58442 32712437 A  ERRORERROR'
2018-11-20 09:05:12 [DEBUG   ] Got command RPT from MCS: ref #567355
2018-11-20 09:05:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:05:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567355  98 58442 32712641 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:05:12 [DEBUG   ] Got command RPT from MCS: ref #567356
2018-11-20 09:05:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:05:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567356  12 58442 32712845 A  ERRORB¢ff'
2018-11-20 09:05:14 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 09:05:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:05:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:05:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:05:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:05:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:05:24 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 09:05:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:05:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:05:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:05:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:05:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:05:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:05:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 09:05:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:05:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:05:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:05:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:05:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:05:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:05:25 [DEBUG   ] Finished updating board flags in 1.804 seconds
2018-11-20 09:05:34 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 09:05:46 [DEBUG   ] Got command RPT from MCS: ref #567357
2018-11-20 09:05:46 [DEBUG   ] summary = ERROR
2018-11-20 09:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567357  13 58442 32746662 A  ERRORERROR'
2018-11-20 09:05:46 [DEBUG   ] Got command RPT from MCS: ref #567358
2018-11-20 09:05:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567358  98 58442 32746662 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:05:54 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 09:06:14 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 09:06:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:06:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:06:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:06:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:06:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:06:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:06:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:06:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:06:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:06:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:06:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:06:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:06:25 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 09:06:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:06:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:06:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:06:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:06:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:06:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:06:25 [DEBUG   ] Finished updating board flags in 1.805 seconds
2018-11-20 09:06:34 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 09:06:54 [DEBUG   ] Finished updating ant stats in 0.596 seconds
2018-11-20 09:07:15 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-20 09:07:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:07:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:07:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:07:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:07:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:07:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:07:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:07:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:07:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:07:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:07:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:07:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 09:07:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:07:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:07:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:07:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:07:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:07:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:07:25 [DEBUG   ] Finished updating board flags in 1.797 seconds
2018-11-20 09:07:35 [DEBUG   ] Finished updating ant stats in 1.208 seconds
2018-11-20 09:07:55 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 09:08:15 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:08:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:08:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370161772000000
2018-11-20 09:08:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:08:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:08:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:08:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:08:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370161772000000
2018-11-20 09:08:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 09:08:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:08:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:08:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:08:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:08:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:08:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:08:25 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:08:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 09:08:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:08:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:08:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:08:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:08:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:08:25 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:08:25 [DEBUG   ] Finished updating board flags in 1.611 seconds
2018-11-20 09:08:35 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:08:55 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:09:15 [DEBUG   ] Finished updating ant stats in 0.994 seconds
2018-11-20 09:09:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:09:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370173532000000
2018-11-20 09:09:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:09:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:09:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:09:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:09:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370173532000000
2018-11-20 09:09:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:09:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:09:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:09:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:09:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:09:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:09:25 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:09:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 09:09:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 09:09:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:09:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:09:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:09:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:09:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:09:25 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:09:25 [DEBUG   ] Finished updating board flags in 1.590 seconds
2018-11-20 09:09:35 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:09:55 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 09:10:12 [DEBUG   ] Got command RPT from MCS: ref #567380
2018-11-20 09:10:12 [DEBUG   ] summary = ERROR
2018-11-20 09:10:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567380  13 58442 33012125 A  ERRORERROR'
2018-11-20 09:10:12 [DEBUG   ] Got command RPT from MCS: ref #567381
2018-11-20 09:10:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:10:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567381  71 58442 33012318 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:10:12 [DEBUG   ] Got command RPT from MCS: ref #567382
2018-11-20 09:10:12 [DEBUG   ] summary = ERROR
2018-11-20 09:10:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567382  13 58442 33012521 A  ERRORERROR'
2018-11-20 09:10:12 [DEBUG   ] Got command RPT from MCS: ref #567384
2018-11-20 09:10:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:10:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567384  71 58442 33012725 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:10:12 [DEBUG   ] Got command RPT from MCS: ref #567385
2018-11-20 09:10:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:10:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567385  12 58442 33012918 A  ERRORB¡ff'
2018-11-20 09:10:15 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 09:10:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:10:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370185292000000
2018-11-20 09:10:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:10:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:10:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:10:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:10:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370185292000000
2018-11-20 09:10:24 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:10:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 09:10:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:10:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:10:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:10:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:10:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:10:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:10:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 09:10:25 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:10:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:10:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:10:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:10:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:10:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:10:25 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:10:25 [DEBUG   ] Finished updating board flags in 1.418 seconds
2018-11-20 09:10:35 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 09:10:46 [DEBUG   ] Got command RPT from MCS: ref #567394
2018-11-20 09:10:46 [DEBUG   ] summary = ERROR
2018-11-20 09:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567394  13 58442 33046653 A  ERRORERROR'
2018-11-20 09:10:46 [DEBUG   ] Got command RPT from MCS: ref #567395
2018-11-20 09:10:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567395  71 58442 33046654 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:10:55 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 09:11:15 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 09:11:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:11:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370197052000000
2018-11-20 09:11:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:11:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:11:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:11:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:11:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370197052000000
2018-11-20 09:11:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 09:11:24 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 09:11:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:11:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:11:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:11:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:11:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:11:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:11:25 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:11:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:11:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:11:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:11:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:11:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:11:25 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:11:25 [DEBUG   ] Finished updating board flags in 1.408 seconds
2018-11-20 09:11:35 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 09:11:55 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 09:12:16 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 09:12:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:12:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370208812000000
2018-11-20 09:12:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:12:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:12:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:12:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:12:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370208812000000
2018-11-20 09:12:24 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:12:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:12:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:12:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:12:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:12:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:12:25 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:12:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 09:12:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:12:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:12:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:12:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:12:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:12:25 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:12:25 [DEBUG   ] Finished updating board flags in 1.384 seconds
2018-11-20 09:12:36 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 09:12:56 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:13:16 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 09:13:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:13:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370220572000000
2018-11-20 09:13:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:13:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:13:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:13:24 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:13:24 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370220572000000
2018-11-20 09:13:24 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:13:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:13:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:13:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302370220768000000
2018-11-20 09:13:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:13:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:13:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:13:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:13:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302370220768000000
2018-11-20 09:13:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 09:13:25 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:13:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:13:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:13:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:13:25 [WARNING ] (13, 1, 'bot')
2018-11-20 09:13:25 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:13:25 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 09:13:25 [DEBUG   ] Finished updating board flags in 1.362 seconds
2018-11-20 09:13:36 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 09:13:56 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 09:14:16 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 09:14:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:14:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:14:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:14:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:14:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:14:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:14:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:14:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:14:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:14:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:14:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:14:26 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:14:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 09:14:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:14:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:14:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:14:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:14:26 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:14:26 [DEBUG   ] Finished updating board flags in 2.051 seconds
2018-11-20 09:14:36 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:14:56 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 09:15:12 [DEBUG   ] Got command RPT from MCS: ref #567415
2018-11-20 09:15:12 [DEBUG   ] summary = ERROR
2018-11-20 09:15:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567415  13 58442 33312205 A  ERRORERROR'
2018-11-20 09:15:12 [DEBUG   ] Got command RPT from MCS: ref #567416
2018-11-20 09:15:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:15:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567416  98 58442 33312397 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:15:12 [DEBUG   ] Got command RPT from MCS: ref #567417
2018-11-20 09:15:12 [DEBUG   ] summary = ERROR
2018-11-20 09:15:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567417  13 58442 33312601 A  ERRORERROR'
2018-11-20 09:15:12 [DEBUG   ] Got command RPT from MCS: ref #567418
2018-11-20 09:15:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:15:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567418  98 58442 33312805 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:15:12 [DEBUG   ] Got command RPT from MCS: ref #567419
2018-11-20 09:15:12 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:15:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567419  12 58442 33312997 A  ERRORB ff'
2018-11-20 09:15:16 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 09:15:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:15:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:15:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:15:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:15:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:15:25 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:15:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:15:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:15:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:15:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:15:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:15:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:15:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 09:15:26 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:15:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:15:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:15:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:15:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:15:26 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:15:26 [DEBUG   ] Finished updating board flags in 1.988 seconds
2018-11-20 09:15:36 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 09:15:46 [DEBUG   ] Got command RPT from MCS: ref #567420
2018-11-20 09:15:46 [DEBUG   ] summary = ERROR
2018-11-20 09:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567420  13 58442 33346669 A  ERRORERROR'
2018-11-20 09:15:46 [DEBUG   ] Got command RPT from MCS: ref #567421
2018-11-20 09:15:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567421  98 58442 33346670 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:15:56 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 09:16:16 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 09:16:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:16:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:16:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:16:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:16:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:16:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 09:16:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:16:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:16:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:16:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:16:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:16:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:16:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 09:16:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 09:16:26 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:16:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 09:16:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:16:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:16:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:16:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:16:26 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:16:26 [DEBUG   ] Finished updating board flags in 1.819 seconds
2018-11-20 09:16:36 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 09:16:56 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 09:17:17 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 09:17:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:17:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:17:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:17:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:17:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:17:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:17:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:17:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:17:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:17:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:17:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:17:26 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:17:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:17:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:17:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:17:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:17:26 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:17:26 [DEBUG   ] Finished updating board flags in 1.803 seconds
2018-11-20 09:17:37 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 09:17:57 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 09:18:17 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 09:18:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:18:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:18:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:18:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:18:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:18:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:18:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:18:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:18:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:18:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:18:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:18:26 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:18:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 09:18:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:18:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:18:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:18:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:18:26 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:18:26 [DEBUG   ] Finished updating board flags in 1.789 seconds
2018-11-20 09:18:37 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:18:57 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 09:19:17 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 09:19:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:19:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370291328000000
2018-11-20 09:19:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:19:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:19:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:19:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:19:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370291328000000
2018-11-20 09:19:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:19:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 09:19:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:19:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:19:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:19:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:19:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:19:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:19:26 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:19:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:19:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:19:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:19:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:19:26 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:19:26 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:19:26 [DEBUG   ] Finished updating board flags in 1.700 seconds
2018-11-20 09:19:37 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:19:57 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:20:12 [DEBUG   ] Got command RPT from MCS: ref #567447
2018-11-20 09:20:12 [DEBUG   ] summary = ERROR
2018-11-20 09:20:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567447  13 58442 33612281 A  ERRORERROR'
2018-11-20 09:20:12 [DEBUG   ] Got command RPT from MCS: ref #567448
2018-11-20 09:20:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:20:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567448  71 58442 33612485 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:20:12 [DEBUG   ] Got command RPT from MCS: ref #567449
2018-11-20 09:20:12 [DEBUG   ] summary = ERROR
2018-11-20 09:20:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567449  13 58442 33612677 A  ERRORERROR'
2018-11-20 09:20:12 [DEBUG   ] Got command RPT from MCS: ref #567451
2018-11-20 09:20:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:20:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567451  71 58442 33612881 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:20:13 [DEBUG   ] Got command RPT from MCS: ref #567452
2018-11-20 09:20:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:20:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567452  12 58442 33613085 A  ERRORB¢ff'
2018-11-20 09:20:17 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:20:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:20:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370303088000000
2018-11-20 09:20:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:20:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:20:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:20:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:20:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370303088000000
2018-11-20 09:20:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 09:20:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 09:20:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:20:25 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 09:20:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:20:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:20:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:20:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:20:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:20:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 09:20:26 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:20:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:20:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:20:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:20:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:20:26 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:20:26 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:20:26 [DEBUG   ] Finished updating board flags in 1.505 seconds
2018-11-20 09:20:37 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:20:46 [DEBUG   ] Got command RPT from MCS: ref #567457
2018-11-20 09:20:46 [DEBUG   ] summary = ERROR
2018-11-20 09:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567457  13 58442 33646662 A  ERRORERROR'
2018-11-20 09:20:46 [DEBUG   ] Got command RPT from MCS: ref #567458
2018-11-20 09:20:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567458  71 58442 33646662 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:20:57 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:21:17 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:21:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:21:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370314848000000
2018-11-20 09:21:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:21:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:21:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:21:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:21:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370314848000000
2018-11-20 09:21:25 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:21:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:21:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302370315044000000
2018-11-20 09:21:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:21:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:21:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:21:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:21:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302370315044000000
2018-11-20 09:21:26 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:21:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:21:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:21:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:21:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:21:26 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:21:26 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 09:21:26 [DEBUG   ] Finished updating board flags in 1.526 seconds
2018-11-20 09:21:37 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:21:57 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:22:17 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:22:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:22:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370326608000000
2018-11-20 09:22:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:22:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:22:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:22:25 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:22:25 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370326608000000
2018-11-20 09:22:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:22:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:22:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302370326804000000
2018-11-20 09:22:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:22:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:22:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:22:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:22:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302370326804000000
2018-11-20 09:22:26 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:22:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 09:22:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:22:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:22:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:22:26 [WARNING ] (13, 1, 'bot')
2018-11-20 09:22:26 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:22:26 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 09:22:26 [DEBUG   ] Finished updating board flags in 1.505 seconds
2018-11-20 09:22:37 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:22:57 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:23:17 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:23:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:23:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:23:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:23:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:23:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:23:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:23:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:23:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:23:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:23:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:23:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:23:27 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:23:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:23:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:23:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:23:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:23:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:23:27 [DEBUG   ] Finished updating board flags in 2.230 seconds
2018-11-20 09:23:37 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:23:57 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:24:17 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:24:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:24:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:24:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:24:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:24:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:24:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:24:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:24:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:24:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:24:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:24:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:24:27 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:24:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:24:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:24:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:24:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:24:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:24:27 [DEBUG   ] Finished updating board flags in 2.233 seconds
2018-11-20 09:24:37 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 09:24:57 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 09:25:12 [DEBUG   ] Got command RPT from MCS: ref #567478
2018-11-20 09:25:12 [DEBUG   ] summary = ERROR
2018-11-20 09:25:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567478  13 58442 33912361 A  ERRORERROR'
2018-11-20 09:25:12 [DEBUG   ] Got command RPT from MCS: ref #567479
2018-11-20 09:25:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:25:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567479  98 58442 33912565 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:25:12 [DEBUG   ] Got command RPT from MCS: ref #567480
2018-11-20 09:25:12 [DEBUG   ] summary = ERROR
2018-11-20 09:25:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567480  13 58442 33912757 A  ERRORERROR'
2018-11-20 09:25:12 [DEBUG   ] Got command RPT from MCS: ref #567481
2018-11-20 09:25:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:25:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567481  98 58442 33912961 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:25:13 [DEBUG   ] Got command RPT from MCS: ref #567482
2018-11-20 09:25:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:25:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567482  12 58442 33913165 A  ERRORB¢ff'
2018-11-20 09:25:17 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:25:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:25:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:25:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:25:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:25:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:25:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:25:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:25:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:25:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:25:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:25:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:25:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:25:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 09:25:27 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:25:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:25:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:25:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:25:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:25:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:25:27 [DEBUG   ] Finished updating board flags in 2.217 seconds
2018-11-20 09:25:37 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 09:25:46 [DEBUG   ] Got command RPT from MCS: ref #567487
2018-11-20 09:25:46 [DEBUG   ] summary = ERROR
2018-11-20 09:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567487  13 58442 33946686 A  ERRORERROR'
2018-11-20 09:25:46 [DEBUG   ] Got command RPT from MCS: ref #567488
2018-11-20 09:25:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567488  98 58442 33946686 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:25:57 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 09:26:17 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 09:26:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:26:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:26:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:26:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:26:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:26:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 09:26:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 09:26:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:26:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:26:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:26:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:26:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:26:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:26:27 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:26:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:26:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:26:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:26:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:26:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:26:27 [DEBUG   ] Finished updating board flags in 2.098 seconds
2018-11-20 09:26:37 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 09:26:57 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 09:27:17 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 09:27:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:27:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:27:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:27:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:27:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:27:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:27:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:27:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:27:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:27:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:27:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:27:27 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:27:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:27:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:27:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:27:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:27:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:27:27 [DEBUG   ] Finished updating board flags in 2.039 seconds
2018-11-20 09:27:37 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 09:27:57 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 09:28:17 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 09:28:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:28:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:28:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:28:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:28:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:28:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:28:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:28:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:28:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:28:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:28:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:28:27 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:28:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:28:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:28:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:28:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:28:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:28:27 [DEBUG   ] Finished updating board flags in 2.016 seconds
2018-11-20 09:28:37 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 09:28:57 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 09:29:17 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 09:29:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:29:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:29:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:29:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:29:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:29:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 09:29:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:29:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:29:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:29:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:29:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:29:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:29:27 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:29:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:29:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:29:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:29:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:29:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:29:27 [DEBUG   ] Finished updating board flags in 1.964 seconds
2018-11-20 09:29:37 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 09:29:57 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 09:30:12 [DEBUG   ] Got command RPT from MCS: ref #567510
2018-11-20 09:30:12 [DEBUG   ] summary = ERROR
2018-11-20 09:30:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567510  13 58442 34212445 A  ERRORERROR'
2018-11-20 09:30:12 [DEBUG   ] Got command RPT from MCS: ref #567511
2018-11-20 09:30:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:30:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567511  98 58442 34212637 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:30:12 [DEBUG   ] Got command RPT from MCS: ref #567512
2018-11-20 09:30:12 [DEBUG   ] summary = ERROR
2018-11-20 09:30:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567512  13 58442 34212841 A  ERRORERROR'
2018-11-20 09:30:13 [DEBUG   ] Got command RPT from MCS: ref #567514
2018-11-20 09:30:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:30:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567514  98 58442 34213045 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:30:13 [DEBUG   ] Got command RPT from MCS: ref #567515
2018-11-20 09:30:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:30:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567515  12 58442 34213249 A  ERRORB¡ff'
2018-11-20 09:30:18 [DEBUG   ] Finished updating ant stats in 1.400 seconds
2018-11-20 09:30:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:30:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:30:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:30:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:30:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:30:26 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 09:30:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:30:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:30:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:30:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:30:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:30:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:30:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 09:30:27 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:30:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:30:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:30:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:30:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:30:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:30:27 [DEBUG   ] Finished updating board flags in 1.800 seconds
2018-11-20 09:30:38 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 09:30:46 [DEBUG   ] Got command RPT from MCS: ref #567520
2018-11-20 09:30:46 [DEBUG   ] summary = ERROR
2018-11-20 09:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567520  13 58442 34246669 A  ERRORERROR'
2018-11-20 09:30:46 [DEBUG   ] Got command RPT from MCS: ref #567521
2018-11-20 09:30:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567521  98 58442 34246670 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:30:58 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 09:31:18 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 09:31:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:31:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370432644000000
2018-11-20 09:31:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:31:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:31:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:31:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:31:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370432644000000
2018-11-20 09:31:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:31:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:31:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:31:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:31:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:31:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:31:27 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:31:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:31:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:31:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:31:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:31:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:31:27 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:31:27 [DEBUG   ] Finished updating board flags in 1.591 seconds
2018-11-20 09:31:38 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:31:58 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 09:32:18 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 09:32:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:32:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370444404000000
2018-11-20 09:32:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:32:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:32:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:32:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:32:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370444404000000
2018-11-20 09:32:26 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:32:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:32:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:32:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:32:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:32:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:32:27 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:32:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:32:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:32:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:32:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:32:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:32:27 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:32:27 [DEBUG   ] Finished updating board flags in 1.536 seconds
2018-11-20 09:32:38 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 09:32:58 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:33:18 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:33:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:33:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370456164000000
2018-11-20 09:33:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:33:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:33:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:33:26 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:33:26 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370456164000000
2018-11-20 09:33:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:33:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:33:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302370456360000000
2018-11-20 09:33:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:33:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:33:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:33:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:33:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302370456360000000
2018-11-20 09:33:27 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:33:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:33:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:33:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:33:27 [WARNING ] (13, 1, 'bot')
2018-11-20 09:33:27 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:33:27 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 09:33:27 [DEBUG   ] Finished updating board flags in 1.573 seconds
2018-11-20 09:33:38 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 09:33:58 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 09:34:18 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:34:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:34:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:34:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:34:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:34:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:34:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 09:34:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:34:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:34:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:34:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:34:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:34:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:34:28 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:34:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 09:34:28 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 09:34:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:34:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:34:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:34:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:34:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:34:28 [DEBUG   ] Finished updating board flags in 2.209 seconds
2018-11-20 09:34:38 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 09:34:58 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:35:12 [DEBUG   ] Got command RPT from MCS: ref #567545
2018-11-20 09:35:12 [DEBUG   ] summary = ERROR
2018-11-20 09:35:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567545  13 58442 34512517 A  ERRORERROR'
2018-11-20 09:35:12 [DEBUG   ] Got command RPT from MCS: ref #567546
2018-11-20 09:35:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:35:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567546  98 58442 34512721 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:35:12 [DEBUG   ] Got command RPT from MCS: ref #567547
2018-11-20 09:35:12 [DEBUG   ] summary = ERROR
2018-11-20 09:35:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567547  13 58442 34512925 A  ERRORERROR'
2018-11-20 09:35:13 [DEBUG   ] Got command RPT from MCS: ref #567548
2018-11-20 09:35:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:35:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567548  98 58442 34513117 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:35:13 [DEBUG   ] Got command RPT from MCS: ref #567549
2018-11-20 09:35:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:35:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567549  12 58442 34513321 A  ERRORB¡ff'
2018-11-20 09:35:18 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:35:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:35:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:35:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:35:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:35:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:35:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 09:35:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:35:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:35:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:35:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:35:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:35:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:35:28 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:35:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 09:35:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:35:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:35:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:35:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:35:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:35:28 [DEBUG   ] Finished updating board flags in 2.198 seconds
2018-11-20 09:35:38 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:35:46 [DEBUG   ] Got command RPT from MCS: ref #567550
2018-11-20 09:35:46 [DEBUG   ] summary = ERROR
2018-11-20 09:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567550  13 58442 34546661 A  ERRORERROR'
2018-11-20 09:35:46 [DEBUG   ] Got command RPT from MCS: ref #567551
2018-11-20 09:35:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567551  98 58442 34546662 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:35:58 [DEBUG   ] Finished updating ant stats in 1.000 seconds
2018-11-20 09:36:18 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 09:36:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:36:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:36:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:36:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:36:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:36:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 09:36:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 09:36:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:36:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:36:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:36:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:36:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:36:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:36:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:36:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 09:36:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:36:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:36:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:36:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:36:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:36:28 [DEBUG   ] Finished updating board flags in 1.906 seconds
2018-11-20 09:36:38 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:36:58 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:37:18 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:37:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:37:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:37:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:37:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:37:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:37:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 09:37:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 09:37:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:37:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:37:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:37:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:37:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:37:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:37:28 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:37:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:37:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:37:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:37:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:37:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:37:28 [DEBUG   ] Finished updating board flags in 1.821 seconds
2018-11-20 09:37:38 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 09:37:58 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:38:18 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 09:38:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:38:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:38:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:38:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:38:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:38:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 09:38:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:38:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:38:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:38:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:38:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:38:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:38:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:38:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:38:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:38:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:38:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:38:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:38:28 [DEBUG   ] Finished updating board flags in 1.776 seconds
2018-11-20 09:38:38 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 09:38:58 [DEBUG   ] Finished updating ant stats in 0.597 seconds
2018-11-20 09:39:18 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 09:39:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:39:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370526920000000
2018-11-20 09:39:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:39:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:39:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:39:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:39:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370526920000000
2018-11-20 09:39:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 09:39:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 09:39:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:39:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:39:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:39:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:39:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:39:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:39:28 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:39:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa28, chip 5
2018-11-20 09:39:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:39:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:39:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:39:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:39:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:39:28 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:39:28 [DEBUG   ] Finished updating board flags in 1.616 seconds
2018-11-20 09:39:38 [DEBUG   ] Finished updating ant stats in 0.404 seconds
2018-11-20 09:39:59 [DEBUG   ] Finished updating ant stats in 1.394 seconds
2018-11-20 09:40:12 [DEBUG   ] Got command RPT from MCS: ref #567573
2018-11-20 09:40:12 [DEBUG   ] summary = ERROR
2018-11-20 09:40:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567573  13 58442 34812597 A  ERRORERROR'
2018-11-20 09:40:12 [DEBUG   ] Got command RPT from MCS: ref #567574
2018-11-20 09:40:12 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:40:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567574  71 58442 34812797 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:40:13 [DEBUG   ] Got command RPT from MCS: ref #567575
2018-11-20 09:40:13 [DEBUG   ] summary = ERROR
2018-11-20 09:40:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567575  13 58442 34813001 A  ERRORERROR'
2018-11-20 09:40:13 [DEBUG   ] Got command RPT from MCS: ref #567577
2018-11-20 09:40:13 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:40:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567577  71 58442 34813205 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:40:13 [DEBUG   ] Got command RPT from MCS: ref #567579
2018-11-20 09:40:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:40:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567579  12 58442 34813397 A  ERRORB£ff'
2018-11-20 09:40:19 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 09:40:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:40:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370538680000000
2018-11-20 09:40:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:40:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:40:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:40:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:40:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370538680000000
2018-11-20 09:40:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:40:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:40:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:40:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:40:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:40:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:40:28 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:40:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:40:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:40:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:40:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:40:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:40:28 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:40:28 [DEBUG   ] Finished updating board flags in 1.594 seconds
2018-11-20 09:40:39 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 09:40:46 [DEBUG   ] Got command RPT from MCS: ref #567587
2018-11-20 09:40:46 [DEBUG   ] summary = ERROR
2018-11-20 09:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567587  13 58442 34846665 A  ERRORERROR'
2018-11-20 09:40:46 [DEBUG   ] Got command RPT from MCS: ref #567588
2018-11-20 09:40:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567588  71 58442 34846666 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:40:59 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 09:41:19 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:41:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:41:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370550440000000
2018-11-20 09:41:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:41:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:41:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:41:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:41:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370550440000000
2018-11-20 09:41:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 09:41:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:41:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:41:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:41:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:41:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:41:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:41:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 09:41:28 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:41:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:41:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:41:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:41:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:41:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:41:28 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:41:28 [DEBUG   ] Finished updating board flags in 1.412 seconds
2018-11-20 09:41:39 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:41:59 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:42:19 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 09:42:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:42:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370562200000000
2018-11-20 09:42:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:42:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:42:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:42:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:42:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370562200000000
2018-11-20 09:42:27 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 09:42:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:42:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:42:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:42:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:42:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:42:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:42:28 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:42:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 09:42:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:42:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:42:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:42:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:42:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:42:28 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:42:28 [DEBUG   ] Finished updating board flags in 1.412 seconds
2018-11-20 09:42:39 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 09:42:59 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 09:43:19 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:43:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:43:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370573960000000
2018-11-20 09:43:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:43:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:43:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:43:27 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:43:27 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370573960000000
2018-11-20 09:43:27 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:43:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:43:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:43:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:43:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:43:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:43:28 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:43:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 09:43:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:43:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:43:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:43:28 [WARNING ] (13, 1, 'bot')
2018-11-20 09:43:28 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:43:28 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:43:28 [DEBUG   ] Finished updating board flags in 1.388 seconds
2018-11-20 09:43:39 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 09:43:59 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 09:44:19 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 09:44:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:44:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:44:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:44:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:44:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:44:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:44:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:44:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:44:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:44:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:44:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:44:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 09:44:29 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:44:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:44:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:44:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:44:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:44:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:44:29 [DEBUG   ] Finished updating board flags in 2.198 seconds
2018-11-20 09:44:39 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 09:44:59 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 09:45:12 [DEBUG   ] Got command RPT from MCS: ref #567608
2018-11-20 09:45:12 [DEBUG   ] summary = ERROR
2018-11-20 09:45:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567608  13 58442 35112685 A  ERRORERROR'
2018-11-20 09:45:12 [DEBUG   ] Got command RPT from MCS: ref #567609
2018-11-20 09:45:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:45:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567609  98 58442 35112889 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:45:13 [DEBUG   ] Got command RPT from MCS: ref #567610
2018-11-20 09:45:13 [DEBUG   ] summary = ERROR
2018-11-20 09:45:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567610  13 58442 35113081 A  ERRORERROR'
2018-11-20 09:45:13 [DEBUG   ] Got command RPT from MCS: ref #567611
2018-11-20 09:45:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:45:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567611  98 58442 35113285 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:45:13 [DEBUG   ] Got command RPT from MCS: ref #567612
2018-11-20 09:45:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:45:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567612  12 58442 35113489 A  ERRORB¢ff'
2018-11-20 09:45:19 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 09:45:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:45:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:45:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:45:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:45:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:45:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 09:45:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:45:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:45:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:45:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:45:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:45:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:45:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 09:45:29 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:45:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 09:45:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:45:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:45:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:45:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:45:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:45:29 [DEBUG   ] Finished updating board flags in 2.006 seconds
2018-11-20 09:45:39 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 09:45:46 [DEBUG   ] Got command RPT from MCS: ref #567613
2018-11-20 09:45:46 [DEBUG   ] summary = ERROR
2018-11-20 09:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567613  13 58442 35146665 A  ERRORERROR'
2018-11-20 09:45:46 [DEBUG   ] Got command RPT from MCS: ref #567614
2018-11-20 09:45:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567614  98 58442 35146666 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:45:59 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 09:46:19 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 09:46:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:46:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:46:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:46:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:46:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:46:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 09:46:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:46:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:46:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:46:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:46:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:46:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:46:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 09:46:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 09:46:29 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:46:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 09:46:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:46:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:46:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:46:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:46:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:46:29 [DEBUG   ] Finished updating board flags in 2.052 seconds
2018-11-20 09:46:39 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 09:46:59 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 09:47:20 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 09:47:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:47:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:47:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:47:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:47:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:47:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:47:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:47:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:47:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:47:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:47:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:47:29 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:47:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:47:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:47:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:47:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:47:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:47:29 [DEBUG   ] Finished updating board flags in 1.944 seconds
2018-11-20 09:47:40 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 09:48:00 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:48:20 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:48:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:48:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:48:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:48:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:48:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:48:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:48:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:48:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:48:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:48:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:48:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:48:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:48:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 09:48:29 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:48:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:48:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:48:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:48:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:48:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:48:29 [DEBUG   ] Finished updating board flags in 1.819 seconds
2018-11-20 09:48:40 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:49:00 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 09:49:20 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:49:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:49:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:49:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:49:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:49:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:49:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 09:49:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 09:49:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:49:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:49:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:49:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:49:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:49:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:49:29 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:49:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:49:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:49:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:49:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:49:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:49:29 [DEBUG   ] Finished updating board flags in 1.836 seconds
2018-11-20 09:49:40 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:50:00 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:50:12 [DEBUG   ] Got command RPT from MCS: ref #567640
2018-11-20 09:50:12 [DEBUG   ] summary = ERROR
2018-11-20 09:50:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567640  13 58442 35412765 A  ERRORERROR'
2018-11-20 09:50:12 [DEBUG   ] Got command RPT from MCS: ref #567641
2018-11-20 09:50:12 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:50:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567641  98 58442 35412957 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:50:13 [DEBUG   ] Got command RPT from MCS: ref #567642
2018-11-20 09:50:13 [DEBUG   ] summary = ERROR
2018-11-20 09:50:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567642  13 58442 35413161 A  ERRORERROR'
2018-11-20 09:50:13 [DEBUG   ] Got command RPT from MCS: ref #567644
2018-11-20 09:50:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:50:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567644  98 58442 35413365 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:50:13 [DEBUG   ] Got command RPT from MCS: ref #567645
2018-11-20 09:50:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:50:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567645  12 58442 35413569 A  ERRORB¡ff'
2018-11-20 09:50:20 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 09:50:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:50:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:50:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:50:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:50:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:50:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:50:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:50:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:50:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:50:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:50:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:50:29 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:50:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 09:50:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:50:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:50:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:50:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:50:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:50:29 [DEBUG   ] Finished updating board flags in 1.812 seconds
2018-11-20 09:50:40 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:50:46 [DEBUG   ] Got command RPT from MCS: ref #567650
2018-11-20 09:50:46 [DEBUG   ] summary = ERROR
2018-11-20 09:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567650  13 58442 35446661 A  ERRORERROR'
2018-11-20 09:50:46 [DEBUG   ] Got command RPT from MCS: ref #567651
2018-11-20 09:50:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 09:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567651  98 58442 35446662 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 09:51:00 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:51:20 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:51:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:51:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:51:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:51:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:51:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:51:28 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:51:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:51:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:51:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:51:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:51:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:51:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:51:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 09:51:29 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:51:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:51:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:51:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:51:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:51:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:51:29 [DEBUG   ] Finished updating board flags in 1.766 seconds
2018-11-20 09:51:40 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 09:52:00 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:52:20 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 09:52:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:52:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370679996000000
2018-11-20 09:52:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:52:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:52:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:52:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:52:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370679996000000
2018-11-20 09:52:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:52:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:52:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:52:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:52:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:52:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:52:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:52:29 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:52:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:52:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:52:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:52:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:52:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:52:29 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:52:29 [DEBUG   ] Finished updating board flags in 1.661 seconds
2018-11-20 09:52:40 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 09:53:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:53:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:53:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:53:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370691756000000
2018-11-20 09:53:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:53:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:53:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:53:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:53:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370691756000000
2018-11-20 09:53:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:53:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 09:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:53:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:53:29 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:53:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:53:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:53:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:53:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:53:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:53:29 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:53:29 [DEBUG   ] Finished updating board flags in 1.522 seconds
2018-11-20 09:53:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:54:00 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:54:20 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 09:54:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:54:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370703516000000
2018-11-20 09:54:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:54:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:54:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:54:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:54:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370703516000000
2018-11-20 09:54:28 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:54:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:54:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:54:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:54:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:54:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:54:29 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:54:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:54:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:54:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:54:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:54:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:54:29 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 09:54:29 [DEBUG   ] Finished updating board flags in 1.458 seconds
2018-11-20 09:54:40 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:55:00 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:55:12 [DEBUG   ] Got command RPT from MCS: ref #567671
2018-11-20 09:55:12 [DEBUG   ] summary = ERROR
2018-11-20 09:55:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567671  13 58442 35712837 A  ERRORERROR'
2018-11-20 09:55:13 [DEBUG   ] Got command RPT from MCS: ref #567672
2018-11-20 09:55:13 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:55:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567672  71 58442 35713041 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:55:13 [DEBUG   ] Got command RPT from MCS: ref #567673
2018-11-20 09:55:13 [DEBUG   ] summary = ERROR
2018-11-20 09:55:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567673  13 58442 35713245 A  ERRORERROR'
2018-11-20 09:55:13 [DEBUG   ] Got command RPT from MCS: ref #567674
2018-11-20 09:55:13 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 09:55:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567674  71 58442 35713449 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 09:55:13 [DEBUG   ] Got command RPT from MCS: ref #567675
2018-11-20 09:55:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 09:55:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567675  12 58442 35713641 A  ERRORB¢ff'
2018-11-20 09:55:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:55:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:55:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370715276000000
2018-11-20 09:55:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:55:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:55:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:55:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:55:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370715276000000
2018-11-20 09:55:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:55:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:55:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 09:55:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:55:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302370715472000000
2018-11-20 09:55:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:55:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:55:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:55:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:55:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302370715472000000
2018-11-20 09:55:29 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:55:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 09:55:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:55:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:55:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:55:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:55:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:55:29 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 09:55:29 [DEBUG   ] Finished updating board flags in 1.532 seconds
2018-11-20 09:55:40 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 09:55:46 [DEBUG   ] Got command RPT from MCS: ref #567680
2018-11-20 09:55:46 [DEBUG   ] summary = ERROR
2018-11-20 09:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567680  13 58442 35746669 A  ERRORERROR'
2018-11-20 09:55:46 [DEBUG   ] Got command RPT from MCS: ref #567681
2018-11-20 09:55:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 09:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567681  79 58442 35746670 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 09:56:00 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:56:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:56:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:56:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370727036000000
2018-11-20 09:56:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:56:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:56:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:56:28 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:56:28 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370727036000000
2018-11-20 09:56:28 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 09:56:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:56:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:56:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302370727232000000
2018-11-20 09:56:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:56:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:56:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:56:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:56:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302370727232000000
2018-11-20 09:56:29 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:56:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:56:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:56:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:56:29 [WARNING ] (13, 1, 'bot')
2018-11-20 09:56:29 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:56:29 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 09:56:29 [DEBUG   ] Finished updating board flags in 1.542 seconds
2018-11-20 09:56:40 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:57:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 09:57:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:57:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:57:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:57:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:57:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:57:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:57:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 09:57:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:57:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:57:30 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 09:57:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 09:57:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:57:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:57:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:57:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:57:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:57:30 [DEBUG   ] Finished updating board flags in 2.209 seconds
2018-11-20 09:57:40 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:58:00 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:58:20 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 09:58:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:58:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:58:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:58:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:58:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:58:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:58:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:58:30 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:58:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:58:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:58:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:58:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:58:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:58:30 [DEBUG   ] Finished updating board flags in 2.066 seconds
2018-11-20 09:58:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:59:00 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 09:59:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 09:59:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 09:59:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 09:59:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 09:59:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 09:59:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 09:59:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 09:59:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 09:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 09:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 09:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 09:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 09:59:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 09:59:30 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 09:59:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 09:59:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 09:59:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 09:59:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:59:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:59:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:59:30 [WARNING ] (13, 1, 'bot')
2018-11-20 09:59:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 09:59:30 [DEBUG   ] Finished updating board flags in 2.031 seconds
2018-11-20 09:59:40 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:00:00 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:00:12 [DEBUG   ] Got command RPT from MCS: ref #567703
2018-11-20 10:00:12 [DEBUG   ] summary = ERROR
2018-11-20 10:00:12 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567703  13 58442 36012925 A  ERRORERROR'
2018-11-20 10:00:13 [DEBUG   ] Got command RPT from MCS: ref #567704
2018-11-20 10:00:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:00:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567704  98 58442 36013129 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:00:13 [DEBUG   ] Got command RPT from MCS: ref #567705
2018-11-20 10:00:13 [DEBUG   ] summary = ERROR
2018-11-20 10:00:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567705  13 58442 36013321 A  ERRORERROR'
2018-11-20 10:00:13 [DEBUG   ] Got command RPT from MCS: ref #567707
2018-11-20 10:00:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:00:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567707  98 58442 36013525 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:00:13 [DEBUG   ] Got command RPT from MCS: ref #567709
2018-11-20 10:00:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:00:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567709  12 58442 36013729 A  ERRORB¢ff'
2018-11-20 10:00:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:00:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:00:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:00:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:00:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:00:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:00:29 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:00:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:00:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:00:30 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:00:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:00:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:00:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:00:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:00:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:00:30 [DEBUG   ] Finished updating board flags in 1.987 seconds
2018-11-20 10:00:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:00:46 [DEBUG   ] Got command RPT from MCS: ref #567713
2018-11-20 10:00:46 [DEBUG   ] summary = ERROR
2018-11-20 10:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567713  13 58442 36046673 A  ERRORERROR'
2018-11-20 10:00:46 [DEBUG   ] Got command RPT from MCS: ref #567714
2018-11-20 10:00:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567714  98 58442 36046674 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:01:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:01:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:01:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:01:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:01:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:01:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:01:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:01:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:01:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:01:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:01:30 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:01:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 10:01:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:01:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:01:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:01:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:01:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:01:30 [DEBUG   ] Finished updating board flags in 1.803 seconds
2018-11-20 10:01:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:02:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:02:20 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:02:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:02:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:02:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:02:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:02:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:02:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 10:02:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:02:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:02:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:02:30 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:02:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:02:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:02:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:02:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:02:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:02:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:02:30 [DEBUG   ] Finished updating board flags in 1.853 seconds
2018-11-20 10:02:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:03:00 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:03:20 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:03:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:03:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:03:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:03:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:03:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:03:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:03:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:03:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:03:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:03:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:03:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:03:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:03:30 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:03:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:03:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:03:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:03:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:03:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:03:30 [DEBUG   ] Finished updating board flags in 1.824 seconds
2018-11-20 10:03:40 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:04:00 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:04:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:04:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:04:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:04:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:04:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:04:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:04:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 10:04:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:04:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:04:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:04:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:04:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:04:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:04:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:04:30 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:04:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:04:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:04:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:04:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:04:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:04:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:04:30 [DEBUG   ] Finished updating board flags in 1.822 seconds
2018-11-20 10:04:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:05:00 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:05:13 [DEBUG   ] Got command RPT from MCS: ref #567738
2018-11-20 10:05:13 [DEBUG   ] summary = ERROR
2018-11-20 10:05:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567738  13 58442 36313001 A  ERRORERROR'
2018-11-20 10:05:13 [DEBUG   ] Got command RPT from MCS: ref #567739
2018-11-20 10:05:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:05:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567739  98 58442 36313205 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:05:13 [DEBUG   ] Got command RPT from MCS: ref #567740
2018-11-20 10:05:13 [DEBUG   ] summary = ERROR
2018-11-20 10:05:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567740  13 58442 36313409 A  ERRORERROR'
2018-11-20 10:05:13 [DEBUG   ] Got command RPT from MCS: ref #567741
2018-11-20 10:05:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:05:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567741  98 58442 36313601 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:05:13 [DEBUG   ] Got command RPT from MCS: ref #567742
2018-11-20 10:05:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:05:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567742  12 58442 36313805 A  ERRORB¢ff'
2018-11-20 10:05:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:05:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:05:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:05:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:05:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:05:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:05:29 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 10:05:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:05:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:05:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:05:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:05:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:05:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:05:30 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:05:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:05:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:05:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:05:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:05:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:05:30 [DEBUG   ] Finished updating board flags in 1.708 seconds
2018-11-20 10:05:40 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:05:46 [DEBUG   ] Got command RPT from MCS: ref #567743
2018-11-20 10:05:46 [DEBUG   ] summary = ERROR
2018-11-20 10:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567743  13 58442 36346673 A  ERRORERROR'
2018-11-20 10:05:46 [DEBUG   ] Got command RPT from MCS: ref #567744
2018-11-20 10:05:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567744  98 58442 36346674 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:06:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:06:20 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:06:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:06:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370844832000000
2018-11-20 10:06:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:06:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:06:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:06:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:06:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370844832000000
2018-11-20 10:06:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:06:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:06:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:06:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:06:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:06:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:06:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:06:30 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:06:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 10:06:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:06:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:06:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:06:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:06:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:06:30 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:06:30 [DEBUG   ] Finished updating board flags in 1.627 seconds
2018-11-20 10:06:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:07:00 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:07:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:07:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:07:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370856592000000
2018-11-20 10:07:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:07:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:07:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:07:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:07:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370856592000000
2018-11-20 10:07:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:07:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:07:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:07:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:07:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:07:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:07:30 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:07:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:07:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:07:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:07:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:07:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:07:30 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:07:30 [DEBUG   ] Finished updating board flags in 1.610 seconds
2018-11-20 10:07:40 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:08:00 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:08:20 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:08:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:08:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370868352000000
2018-11-20 10:08:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:08:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:08:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:08:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:08:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370868352000000
2018-11-20 10:08:29 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:08:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:08:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:08:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:08:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:08:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:08:30 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:08:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 10:08:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:08:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:08:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:08:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:08:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:08:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:08:30 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:08:30 [DEBUG   ] Finished updating board flags in 1.585 seconds
2018-11-20 10:08:40 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:09:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:09:20 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:09:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:09:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370880112000000
2018-11-20 10:09:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:09:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:09:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:09:29 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:09:29 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370880112000000
2018-11-20 10:09:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:09:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:09:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302370880308000000
2018-11-20 10:09:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:09:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:09:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:09:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:09:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302370880308000000
2018-11-20 10:09:30 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:09:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:09:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:09:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:09:30 [WARNING ] (13, 1, 'bot')
2018-11-20 10:09:30 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:09:30 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 10:09:30 [DEBUG   ] Finished updating board flags in 1.586 seconds
2018-11-20 10:09:40 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:10:00 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:10:13 [DEBUG   ] Got command RPT from MCS: ref #567766
2018-11-20 10:10:13 [DEBUG   ] summary = ERROR
2018-11-20 10:10:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567766  13 58442 36613077 A  ERRORERROR'
2018-11-20 10:10:13 [DEBUG   ] Got command RPT from MCS: ref #567767
2018-11-20 10:10:13 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 10:10:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567767  79 58442 36613281 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 10:10:13 [DEBUG   ] Got command RPT from MCS: ref #567768
2018-11-20 10:10:13 [DEBUG   ] summary = ERROR
2018-11-20 10:10:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567768  13 58442 36613485 A  ERRORERROR'
2018-11-20 10:10:13 [DEBUG   ] Got command RPT from MCS: ref #567770
2018-11-20 10:10:13 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 10:10:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567770  79 58442 36613689 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 10:10:13 [DEBUG   ] Got command RPT from MCS: ref #567771
2018-11-20 10:10:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:10:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567771  12 58442 36613881 A  ERRORB¡ff'
2018-11-20 10:10:20 [DEBUG   ] Finished updating ant stats in 0.997 seconds
2018-11-20 10:10:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:10:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:10:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:10:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:10:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:10:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:10:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:10:31 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:10:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:10:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:10:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:10:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:10:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:10:31 [DEBUG   ] Finished updating board flags in 2.338 seconds
2018-11-20 10:10:40 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:10:46 [DEBUG   ] Got command RPT from MCS: ref #567780
2018-11-20 10:10:46 [DEBUG   ] summary = ERROR
2018-11-20 10:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567780  13 58442 36646673 A  ERRORERROR'
2018-11-20 10:10:46 [DEBUG   ] Got command RPT from MCS: ref #567781
2018-11-20 10:10:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567781  98 58442 36646674 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:11:00 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 10:11:20 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 10:11:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:11:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:11:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:11:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:11:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:11:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:11:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:11:31 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:11:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:11:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:11:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:11:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:11:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:11:31 [DEBUG   ] Finished updating board flags in 2.054 seconds
2018-11-20 10:11:40 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 10:12:00 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 10:12:20 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 10:12:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:12:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:12:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:12:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:12:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:12:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:12:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:12:31 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:12:31 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 10:12:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:12:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:12:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:12:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:12:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:12:31 [DEBUG   ] Finished updating board flags in 2.024 seconds
2018-11-20 10:12:40 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 10:13:00 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 10:13:20 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 10:13:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:13:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:13:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:13:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:13:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:13:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:13:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:13:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:13:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:13:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:13:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:13:31 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:13:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:13:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:13:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:13:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:13:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:13:31 [DEBUG   ] Finished updating board flags in 1.983 seconds
2018-11-20 10:13:41 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 10:14:01 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 10:14:21 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 10:14:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:14:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:14:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:14:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:14:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:14:30 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:14:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:14:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:14:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:14:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:14:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:14:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:14:31 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:14:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:14:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:14:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:14:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:14:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:14:31 [DEBUG   ] Finished updating board flags in 1.834 seconds
2018-11-20 10:14:41 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 10:15:01 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 10:15:13 [DEBUG   ] Got command RPT from MCS: ref #567801
2018-11-20 10:15:13 [DEBUG   ] summary = ERROR
2018-11-20 10:15:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567801  13 58442 36913165 A  ERRORERROR'
2018-11-20 10:15:13 [DEBUG   ] Got command RPT from MCS: ref #567802
2018-11-20 10:15:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:15:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567802  98 58442 36913369 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:15:13 [DEBUG   ] Got command RPT from MCS: ref #567803
2018-11-20 10:15:13 [DEBUG   ] summary = ERROR
2018-11-20 10:15:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567803  13 58442 36913573 A  ERRORERROR'
2018-11-20 10:15:13 [DEBUG   ] Got command RPT from MCS: ref #567804
2018-11-20 10:15:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:15:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567804  98 58442 36913769 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:15:13 [DEBUG   ] Got command RPT from MCS: ref #567805
2018-11-20 10:15:13 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:15:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567805  12 58442 36913973 A  ERRORB¡ff'
2018-11-20 10:15:21 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-20 10:15:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:15:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:15:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:15:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:15:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:15:30 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:15:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 10:15:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:15:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:15:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:15:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:15:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:15:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:15:31 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:15:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:15:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:15:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:15:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:15:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:15:31 [DEBUG   ] Finished updating board flags in 1.810 seconds
2018-11-20 10:15:41 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:15:46 [DEBUG   ] Got command RPT from MCS: ref #567806
2018-11-20 10:15:46 [DEBUG   ] summary = ERROR
2018-11-20 10:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567806  13 58442 36946665 A  ERRORERROR'
2018-11-20 10:15:46 [DEBUG   ] Got command RPT from MCS: ref #567807
2018-11-20 10:15:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567807  98 58442 36946666 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:16:01 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:16:21 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:16:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:16:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:16:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:16:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:16:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:16:30 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:16:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:16:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:16:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:16:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:16:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:16:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:16:31 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:16:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:16:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:16:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:16:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:16:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:16:31 [DEBUG   ] Finished updating board flags in 1.754 seconds
2018-11-20 10:16:41 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:17:01 [DEBUG   ] Finished updating ant stats in 1.071 seconds
2018-11-20 10:17:21 [DEBUG   ] Finished updating ant stats in 0.931 seconds
2018-11-20 10:17:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:17:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370974388000000
2018-11-20 10:17:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:17:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:17:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:17:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:17:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370974388000000
2018-11-20 10:17:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:17:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:17:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:17:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:17:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:17:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:17:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:17:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:17:31 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:17:31 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 10:17:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:17:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:17:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:17:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:17:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:17:31 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:17:31 [DEBUG   ] Finished updating board flags in 1.641 seconds
2018-11-20 10:17:41 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 10:18:01 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 10:18:21 [DEBUG   ] Finished updating ant stats in 0.790 seconds
2018-11-20 10:18:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:18:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370986148000000
2018-11-20 10:18:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:18:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:18:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:18:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:18:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370986148000000
2018-11-20 10:18:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:18:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:18:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:18:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:18:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:18:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:18:31 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:18:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:18:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:18:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:18:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:18:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:18:31 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:18:31 [DEBUG   ] Finished updating board flags in 1.560 seconds
2018-11-20 10:18:41 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 10:19:01 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 10:19:21 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 10:19:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:19:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302370997908000000
2018-11-20 10:19:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:19:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:19:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:19:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:19:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302370997908000000
2018-11-20 10:19:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:19:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:19:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:19:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:19:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:19:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:19:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:19:31 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:19:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:19:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:19:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:19:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:19:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:19:31 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:19:31 [DEBUG   ] Finished updating board flags in 1.419 seconds
2018-11-20 10:19:41 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 10:20:01 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 10:20:13 [DEBUG   ] Got command RPT from MCS: ref #567833
2018-11-20 10:20:13 [DEBUG   ] summary = ERROR
2018-11-20 10:20:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567833  13 58442 37213245 A  ERRORERROR'
2018-11-20 10:20:13 [DEBUG   ] Got command RPT from MCS: ref #567834
2018-11-20 10:20:13 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 10:20:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567834  71 58442 37213449 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 10:20:13 [DEBUG   ] Got command RPT from MCS: ref #567835
2018-11-20 10:20:13 [DEBUG   ] summary = ERROR
2018-11-20 10:20:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567835  13 58442 37213653 A  ERRORERROR'
2018-11-20 10:20:13 [DEBUG   ] Got command RPT from MCS: ref #567837
2018-11-20 10:20:13 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 10:20:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567837  71 58442 37213845 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 10:20:14 [DEBUG   ] Got command RPT from MCS: ref #567838
2018-11-20 10:20:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:20:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567838  12 58442 37214049 A  ERRORB ff'
2018-11-20 10:20:21 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 10:20:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:20:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371009668000000
2018-11-20 10:20:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:20:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:20:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:20:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:20:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371009668000000
2018-11-20 10:20:30 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:20:30 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:20:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:20:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:20:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:20:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:20:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:20:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:20:31 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:20:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:20:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:20:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:20:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:20:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:20:31 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:20:31 [DEBUG   ] Finished updating board flags in 1.479 seconds
2018-11-20 10:20:41 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 10:20:46 [DEBUG   ] Got command RPT from MCS: ref #567843
2018-11-20 10:20:46 [DEBUG   ] summary = ERROR
2018-11-20 10:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567843  13 58442 37246673 A  ERRORERROR'
2018-11-20 10:20:46 [DEBUG   ] Got command RPT from MCS: ref #567844
2018-11-20 10:20:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 10:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567844  71 58442 37246674 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 10:21:01 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 10:21:21 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 10:21:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:21:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371021428000000
2018-11-20 10:21:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:21:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:21:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:21:30 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:21:30 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371021428000000
2018-11-20 10:21:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:21:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:21:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371021624000000
2018-11-20 10:21:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:21:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:21:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:21:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:21:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371021624000000
2018-11-20 10:21:31 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:21:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:21:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:21:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:21:31 [WARNING ] (13, 1, 'bot')
2018-11-20 10:21:31 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:21:31 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 10:21:31 [DEBUG   ] Finished updating board flags in 1.524 seconds
2018-11-20 10:21:41 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 10:22:01 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 10:22:22 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 10:22:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:22:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:22:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:22:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:22:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:22:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:22:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:22:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:22:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:22:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:22:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:22:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:22:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:22:32 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:22:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:22:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:22:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:22:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:22:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:22:32 [DEBUG   ] Finished updating board flags in 2.297 seconds
2018-11-20 10:22:42 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 10:23:02 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 10:23:22 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 10:23:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:23:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:23:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:23:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:23:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:23:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 1
2018-11-20 10:23:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 10:23:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:23:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 10:23:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:23:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:23:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:23:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:23:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:23:32 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:23:32 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 10:23:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:23:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:23:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:23:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:23:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:23:32 [DEBUG   ] Finished updating board flags in 2.099 seconds
2018-11-20 10:23:42 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 10:24:02 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:24:22 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:24:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:24:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:24:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:24:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:24:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:24:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 10:24:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 10:24:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:24:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:24:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:24:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:24:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:24:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:24:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:24:32 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:24:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:24:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:24:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:24:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:24:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:24:32 [DEBUG   ] Finished updating board flags in 2.012 seconds
2018-11-20 10:24:42 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:25:02 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:25:13 [DEBUG   ] Got command RPT from MCS: ref #567864
2018-11-20 10:25:13 [DEBUG   ] summary = ERROR
2018-11-20 10:25:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567864  13 58442 37513329 A  ERRORERROR'
2018-11-20 10:25:13 [DEBUG   ] Got command RPT from MCS: ref #567865
2018-11-20 10:25:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:25:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567865  98 58442 37513521 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:25:13 [DEBUG   ] Got command RPT from MCS: ref #567866
2018-11-20 10:25:13 [DEBUG   ] summary = ERROR
2018-11-20 10:25:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567866  13 58442 37513732 A  ERRORERROR'
2018-11-20 10:25:13 [DEBUG   ] Got command RPT from MCS: ref #567867
2018-11-20 10:25:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:25:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567867  98 58442 37513933 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:25:14 [DEBUG   ] Got command RPT from MCS: ref #567868
2018-11-20 10:25:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:25:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567868  12 58442 37514125 A  ERRORB£ff'
2018-11-20 10:25:22 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:25:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:25:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:25:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:25:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:25:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:25:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-20 10:25:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:25:31 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 10:25:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:25:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:25:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:25:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:25:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:25:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:25:32 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:25:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:25:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:25:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:25:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:25:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:25:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:25:32 [DEBUG   ] Finished updating board flags in 2.022 seconds
2018-11-20 10:25:42 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:25:46 [DEBUG   ] Got command RPT from MCS: ref #567873
2018-11-20 10:25:46 [DEBUG   ] summary = ERROR
2018-11-20 10:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567873  13 58442 37546677 A  ERRORERROR'
2018-11-20 10:25:46 [DEBUG   ] Got command RPT from MCS: ref #567874
2018-11-20 10:25:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567874  98 58442 37546678 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:26:02 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:26:22 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:26:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:26:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:26:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:26:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:26:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:26:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 10:26:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:26:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:26:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:26:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:26:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:26:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:26:32 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:26:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:26:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:26:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:26:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:26:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:26:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:26:32 [DEBUG   ] Finished updating board flags in 1.981 seconds
2018-11-20 10:26:42 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:27:02 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:27:22 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:27:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:27:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:27:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:27:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:27:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:27:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 10:27:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:27:31 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 10:27:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:27:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:27:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:27:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:27:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:27:32 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:27:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:27:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:27:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:27:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:27:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:27:32 [DEBUG   ] Finished updating board flags in 1.782 seconds
2018-11-20 10:27:42 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:28:02 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:28:22 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:28:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:28:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371103944000000
2018-11-20 10:28:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:28:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:28:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:28:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:28:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371103944000000
2018-11-20 10:28:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:28:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:28:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:28:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:28:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:28:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:28:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:28:32 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:28:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 10:28:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:28:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:28:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:28:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:28:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:28:32 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:28:32 [DEBUG   ] Finished updating board flags in 1.702 seconds
2018-11-20 10:28:42 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:29:02 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:29:22 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:29:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:29:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371115704000000
2018-11-20 10:29:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:29:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:29:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:29:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:29:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371115704000000
2018-11-20 10:29:31 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:29:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:29:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:29:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:29:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:29:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:29:32 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:29:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:29:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:29:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:29:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:29:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:29:32 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:29:32 [DEBUG   ] Finished updating board flags in 1.525 seconds
2018-11-20 10:29:42 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:30:02 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:30:13 [DEBUG   ] Got command RPT from MCS: ref #567896
2018-11-20 10:30:13 [DEBUG   ] summary = ERROR
2018-11-20 10:30:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567896  13 58442 37813401 A  ERRORERROR'
2018-11-20 10:30:13 [DEBUG   ] Got command RPT from MCS: ref #567897
2018-11-20 10:30:13 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 10:30:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567897  71 58442 37813605 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 10:30:13 [DEBUG   ] Got command RPT from MCS: ref #567898
2018-11-20 10:30:13 [DEBUG   ] summary = ERROR
2018-11-20 10:30:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567898  13 58442 37813802 A  ERRORERROR'
2018-11-20 10:30:14 [DEBUG   ] Got command RPT from MCS: ref #567899
2018-11-20 10:30:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 10:30:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567899  71 58442 37814005 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 10:30:14 [DEBUG   ] Got command RPT from MCS: ref #567901
2018-11-20 10:30:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:30:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567901  12 58442 37814209 A  ERRORB£ff'
2018-11-20 10:30:22 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:30:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:30:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371127464000000
2018-11-20 10:30:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:30:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:30:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:30:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:30:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371127464000000
2018-11-20 10:30:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:30:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:30:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371127660000000
2018-11-20 10:30:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:30:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:30:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:30:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:30:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371127660000000
2018-11-20 10:30:32 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:30:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:30:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:30:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:30:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:30:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:30:32 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 10:30:32 [DEBUG   ] Finished updating board flags in 1.528 seconds
2018-11-20 10:30:42 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:30:46 [DEBUG   ] Got command RPT from MCS: ref #567906
2018-11-20 10:30:46 [DEBUG   ] summary = ERROR
2018-11-20 10:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567906  13 58442 37846669 A  ERRORERROR'
2018-11-20 10:30:46 [DEBUG   ] Got command RPT from MCS: ref #567907
2018-11-20 10:30:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 10:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567907  79 58442 37846670 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 10:31:02 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:31:22 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:31:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:31:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371139224000000
2018-11-20 10:31:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:31:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:31:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:31:31 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:31:31 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371139224000000
2018-11-20 10:31:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:31:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:31:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371139420000000
2018-11-20 10:31:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:31:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:31:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:31:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:31:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371139420000000
2018-11-20 10:31:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:31:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:31:32 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:31:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:31:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:31:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:31:32 [WARNING ] (13, 1, 'bot')
2018-11-20 10:31:32 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:31:32 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 10:31:32 [DEBUG   ] Finished updating board flags in 1.542 seconds
2018-11-20 10:31:42 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:32:02 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:32:22 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:32:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:32:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:32:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:32:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:32:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:32:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:32:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:32:33 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:32:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:32:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:32:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:32:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:32:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:32:33 [DEBUG   ] Finished updating board flags in 2.188 seconds
2018-11-20 10:32:42 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:33:02 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:33:22 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:33:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:33:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:33:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:33:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:33:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:33:32 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:33:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 10:33:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:33:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:33:33 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:33:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:33:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:33:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:33:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:33:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:33:33 [DEBUG   ] Finished updating board flags in 2.040 seconds
2018-11-20 10:33:42 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:34:02 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:34:22 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 10:34:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:34:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:34:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:34:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:34:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:34:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 10:34:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:34:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:34:33 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:34:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 10:34:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:34:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:34:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:34:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:34:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:34:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:34:33 [DEBUG   ] Finished updating board flags in 2.012 seconds
2018-11-20 10:34:42 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 10:35:02 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 10:35:13 [DEBUG   ] Got command RPT from MCS: ref #567931
2018-11-20 10:35:13 [DEBUG   ] summary = ERROR
2018-11-20 10:35:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567931  13 58442 38113485 A  ERRORERROR'
2018-11-20 10:35:13 [DEBUG   ] Got command RPT from MCS: ref #567932
2018-11-20 10:35:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:35:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567932  98 58442 38113689 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:35:13 [DEBUG   ] Got command RPT from MCS: ref #567933
2018-11-20 10:35:13 [DEBUG   ] summary = ERROR
2018-11-20 10:35:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567933  13 58442 38113881 A  ERRORERROR'
2018-11-20 10:35:14 [DEBUG   ] Got command RPT from MCS: ref #567934
2018-11-20 10:35:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:35:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567934  98 58442 38114085 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:35:14 [DEBUG   ] Got command RPT from MCS: ref #567935
2018-11-20 10:35:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:35:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567935  12 58442 38114289 A  ERRORB¡ff'
2018-11-20 10:35:22 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:35:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:35:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:35:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:35:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:35:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:35:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:35:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:35:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:35:33 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:35:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:35:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:35:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:35:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:35:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:35:33 [DEBUG   ] Finished updating board flags in 1.865 seconds
2018-11-20 10:35:42 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 10:35:46 [DEBUG   ] Got command RPT from MCS: ref #567936
2018-11-20 10:35:46 [DEBUG   ] summary = ERROR
2018-11-20 10:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567936  13 58442 38146677 A  ERRORERROR'
2018-11-20 10:35:46 [DEBUG   ] Got command RPT from MCS: ref #567937
2018-11-20 10:35:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567937  98 58442 38146678 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:36:02 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 10:36:22 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:36:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:36:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:36:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:36:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:36:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:36:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:36:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:36:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:36:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:36:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:36:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:36:33 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:36:33 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:36:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:36:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:36:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:36:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:36:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:36:33 [DEBUG   ] Finished updating board flags in 1.826 seconds
2018-11-20 10:36:42 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:37:02 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:37:22 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:37:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:37:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:37:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:37:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:37:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:37:32 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 10:37:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:37:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:37:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:37:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:37:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:37:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:37:33 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:37:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:37:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:37:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:37:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:37:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:37:33 [DEBUG   ] Finished updating board flags in 1.787 seconds
2018-11-20 10:37:42 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:38:02 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:38:22 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:38:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:38:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371221740000000
2018-11-20 10:38:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:38:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:38:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:38:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:38:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371221740000000
2018-11-20 10:38:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:38:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:38:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:38:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:38:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:38:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:38:33 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:38:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:38:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:38:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:38:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:38:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:38:33 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:38:33 [DEBUG   ] Finished updating board flags in 1.620 seconds
2018-11-20 10:38:42 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:39:02 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 10:39:22 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:39:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:39:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371233500000000
2018-11-20 10:39:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:39:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:39:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:39:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:39:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371233500000000
2018-11-20 10:39:32 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:39:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:39:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:39:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:39:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:39:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:39:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:39:33 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:39:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:39:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:39:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:39:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:39:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:39:33 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:39:33 [DEBUG   ] Finished updating board flags in 1.591 seconds
2018-11-20 10:39:42 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:40:02 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 10:40:13 [DEBUG   ] Got command RPT from MCS: ref #567959
2018-11-20 10:40:13 [DEBUG   ] summary = ERROR
2018-11-20 10:40:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567959  13 58442 38413557 A  ERRORERROR'
2018-11-20 10:40:13 [DEBUG   ] Got command RPT from MCS: ref #567960
2018-11-20 10:40:13 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 10:40:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567960  71 58442 38413761 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 10:40:13 [DEBUG   ] Got command RPT from MCS: ref #567961
2018-11-20 10:40:13 [DEBUG   ] summary = ERROR
2018-11-20 10:40:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567961  13 58442 38413965 A  ERRORERROR'
2018-11-20 10:40:14 [DEBUG   ] Got command RPT from MCS: ref #567962
2018-11-20 10:40:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 10:40:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567962  71 58442 38414169 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 10:40:14 [DEBUG   ] Got command RPT from MCS: ref #567964
2018-11-20 10:40:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:40:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567964  12 58442 38414361 A  ERRORB¡ff'
2018-11-20 10:40:22 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:40:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:40:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371245260000000
2018-11-20 10:40:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:40:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:40:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:40:32 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:40:32 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371245260000000
2018-11-20 10:40:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:40:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:40:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371245456000000
2018-11-20 10:40:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:40:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:40:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:40:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:40:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371245456000000
2018-11-20 10:40:33 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:40:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:40:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:40:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:40:33 [WARNING ] (13, 1, 'bot')
2018-11-20 10:40:33 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:40:33 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 10:40:33 [DEBUG   ] Finished updating board flags in 1.595 seconds
2018-11-20 10:40:42 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 10:40:46 [DEBUG   ] Got command RPT from MCS: ref #567973
2018-11-20 10:40:46 [DEBUG   ] summary = ERROR
2018-11-20 10:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567973  13 58442 38446681 A  ERRORERROR'
2018-11-20 10:40:46 [DEBUG   ] Got command RPT from MCS: ref #567974
2018-11-20 10:40:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 10:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567974  79 58442 38446682 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 10:41:02 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 10:41:22 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 10:41:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:41:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:41:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:41:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:41:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:41:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 10:41:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:41:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:41:34 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:41:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:41:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:41:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:41:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:41:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:41:34 [DEBUG   ] Finished updating board flags in 2.318 seconds
2018-11-20 10:41:42 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 10:42:02 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 10:42:22 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 10:42:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:42:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:42:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:42:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:42:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:42:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:42:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:42:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:42:34 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:42:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:42:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:42:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:42:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:42:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:42:34 [DEBUG   ] Finished updating board flags in 2.080 seconds
2018-11-20 10:42:43 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 10:43:03 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 10:43:23 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 10:43:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:43:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:43:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:43:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:43:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:43:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 10:43:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:43:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:43:34 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:43:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:43:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:43:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:43:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:43:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:43:34 [DEBUG   ] Finished updating board flags in 2.024 seconds
2018-11-20 10:43:43 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 10:44:03 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 10:44:23 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 10:44:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:44:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:44:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:44:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:44:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:44:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:44:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 10:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:44:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:44:34 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:44:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:44:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:44:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:44:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:44:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:44:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:44:34 [DEBUG   ] Finished updating board flags in 1.994 seconds
2018-11-20 10:44:43 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 10:45:03 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 10:45:13 [DEBUG   ] Got command RPT from MCS: ref #567994
2018-11-20 10:45:13 [DEBUG   ] summary = ERROR
2018-11-20 10:45:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567994  13 58442 38713645 A  ERRORERROR'
2018-11-20 10:45:13 [DEBUG   ] Got command RPT from MCS: ref #567995
2018-11-20 10:45:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:45:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567995  98 58442 38713837 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:45:14 [DEBUG   ] Got command RPT from MCS: ref #567996
2018-11-20 10:45:14 [DEBUG   ] summary = ERROR
2018-11-20 10:45:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567996  13 58442 38714041 A  ERRORERROR'
2018-11-20 10:45:14 [DEBUG   ] Got command RPT from MCS: ref #567997
2018-11-20 10:45:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:45:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567997  98 58442 38714245 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:45:14 [DEBUG   ] Got command RPT from MCS: ref #567998
2018-11-20 10:45:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:45:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567998  12 58442 38714449 A  ERRORB¡ff'
2018-11-20 10:45:23 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 10:45:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:45:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:45:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:45:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:45:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:45:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 10:45:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:45:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:45:34 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:45:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 10:45:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:45:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:45:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:45:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:45:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:45:34 [DEBUG   ] Finished updating board flags in 1.815 seconds
2018-11-20 10:45:43 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 10:45:46 [DEBUG   ] Got command RPT from MCS: ref #567999
2018-11-20 10:45:46 [DEBUG   ] summary = ERROR
2018-11-20 10:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   567999  13 58442 38746674 A  ERRORERROR'
2018-11-20 10:45:46 [DEBUG   ] Got command RPT from MCS: ref #568000
2018-11-20 10:45:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568000  98 58442 38746674 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:46:03 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 10:46:23 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 10:46:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:46:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:46:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:46:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:46:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:46:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:46:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:46:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:46:34 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:46:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:46:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:46:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:46:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:46:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:46:34 [DEBUG   ] Finished updating board flags in 1.807 seconds
2018-11-20 10:46:43 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 10:47:03 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 10:47:23 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 10:47:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:47:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:47:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:47:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:47:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:47:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:47:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:47:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:47:34 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:47:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:47:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:47:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:47:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:47:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:47:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:47:34 [DEBUG   ] Finished updating board flags in 1.893 seconds
2018-11-20 10:47:43 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 10:48:03 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 10:48:23 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 10:48:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:48:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:48:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:48:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:48:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:48:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:48:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:48:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:48:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:48:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:48:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:48:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:48:34 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:48:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 10:48:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:48:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:48:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:48:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:48:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:48:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:48:34 [DEBUG   ] Finished updating board flags in 1.821 seconds
2018-11-20 10:48:43 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 10:49:03 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 10:49:23 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 10:49:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:49:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:49:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:49:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:49:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:49:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:49:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 10:49:33 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 10:49:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:49:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:49:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:49:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:49:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:49:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:49:34 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:49:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:49:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:49:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:49:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:49:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:49:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:49:34 [DEBUG   ] Finished updating board flags in 1.816 seconds
2018-11-20 10:49:43 [DEBUG   ] Finished updating ant stats in 1.200 seconds
2018-11-20 10:50:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:50:13 [DEBUG   ] Got command RPT from MCS: ref #568026
2018-11-20 10:50:13 [DEBUG   ] summary = ERROR
2018-11-20 10:50:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568026  13 58442 39013717 A  ERRORERROR'
2018-11-20 10:50:13 [DEBUG   ] Got command RPT from MCS: ref #568027
2018-11-20 10:50:13 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:50:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568027  98 58442 39013921 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:50:14 [DEBUG   ] Got command RPT from MCS: ref #568028
2018-11-20 10:50:14 [DEBUG   ] summary = ERROR
2018-11-20 10:50:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568028  13 58442 39014125 A  ERRORERROR'
2018-11-20 10:50:14 [DEBUG   ] Got command RPT from MCS: ref #568030
2018-11-20 10:50:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:50:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568030  98 58442 39014329 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:50:14 [DEBUG   ] Got command RPT from MCS: ref #568031
2018-11-20 10:50:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:50:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568031  12 58442 39014521 A  ERRORB£ff'
2018-11-20 10:50:23 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:50:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:50:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:50:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:50:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:50:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:50:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:50:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 10:50:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 10:50:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:50:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:50:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:50:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:50:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:50:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:50:34 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:50:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:50:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:50:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:50:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:50:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:50:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:50:34 [DEBUG   ] Finished updating board flags in 1.650 seconds
2018-11-20 10:50:43 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:50:46 [DEBUG   ] Got command RPT from MCS: ref #568036
2018-11-20 10:50:46 [DEBUG   ] summary = ERROR
2018-11-20 10:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568036  13 58442 39046677 A  ERRORERROR'
2018-11-20 10:50:46 [DEBUG   ] Got command RPT from MCS: ref #568037
2018-11-20 10:50:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 10:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568037  98 58442 39046678 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 10:51:03 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:51:23 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:51:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:51:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371374816000000
2018-11-20 10:51:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:51:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:51:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:51:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:51:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371374816000000
2018-11-20 10:51:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:51:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:51:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:51:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:51:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:51:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:51:34 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:51:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:51:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:51:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:51:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:51:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:51:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:51:34 [DEBUG   ] Finished updating board flags in 1.615 seconds
2018-11-20 10:51:43 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:52:03 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:52:23 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:52:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:52:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371386576000000
2018-11-20 10:52:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:52:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:52:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:52:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:52:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371386576000000
2018-11-20 10:52:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:52:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:52:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:52:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:52:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:52:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:52:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:52:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:52:34 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:52:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:52:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:52:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:52:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:52:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:52:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:52:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:52:34 [DEBUG   ] Finished updating board flags in 1.617 seconds
2018-11-20 10:52:43 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:53:03 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:53:23 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:53:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:53:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371398336000000
2018-11-20 10:53:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:53:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:53:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:53:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:53:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371398336000000
2018-11-20 10:53:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:53:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:53:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:53:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:53:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:53:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:53:34 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:53:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 10:53:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:53:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:53:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:53:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:53:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:53:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 10:53:34 [DEBUG   ] Finished updating board flags in 1.576 seconds
2018-11-20 10:53:43 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:54:03 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 10:54:23 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 10:54:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:54:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371410096000000
2018-11-20 10:54:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:54:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:54:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:54:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:54:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371410096000000
2018-11-20 10:54:33 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 10:54:33 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:54:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:54:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:54:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:54:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:54:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:54:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371410292000000
2018-11-20 10:54:34 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:54:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:54:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:54:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:54:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:54:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:54:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #1
2018-11-20 10:54:34 [DEBUG   ] Finished updating board flags in 1.563 seconds
2018-11-20 10:54:43 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 10:55:03 [DEBUG   ] Finished updating ant stats in 0.995 seconds
2018-11-20 10:55:13 [DEBUG   ] Got command RPT from MCS: ref #568057
2018-11-20 10:55:13 [DEBUG   ] summary = ERROR
2018-11-20 10:55:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568057  13 58442 39313801 A  ERRORERROR'
2018-11-20 10:55:14 [DEBUG   ] Got command RPT from MCS: ref #568058
2018-11-20 10:55:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #1
2018-11-20 10:55:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568058  75 58442 39314005 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #1'
2018-11-20 10:55:14 [DEBUG   ] Got command RPT from MCS: ref #568059
2018-11-20 10:55:14 [DEBUG   ] summary = ERROR
2018-11-20 10:55:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568059  13 58442 39314209 A  ERRORERROR'
2018-11-20 10:55:14 [DEBUG   ] Got command RPT from MCS: ref #568060
2018-11-20 10:55:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #1
2018-11-20 10:55:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568060  75 58442 39314401 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #1'
2018-11-20 10:55:14 [DEBUG   ] Got command RPT from MCS: ref #568061
2018-11-20 10:55:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 10:55:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568061  12 58442 39314605 A  ERRORB¢ff'
2018-11-20 10:55:23 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 10:55:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:55:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371421856000000
2018-11-20 10:55:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:55:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:55:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:55:33 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:55:33 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371421856000000
2018-11-20 10:55:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:55:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:55:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371422052000000
2018-11-20 10:55:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:55:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:55:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:55:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:55:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371422052000000
2018-11-20 10:55:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:55:34 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 10:55:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 10:55:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:55:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:55:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:55:34 [WARNING ] (13, 1, 'bot')
2018-11-20 10:55:34 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:55:34 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 10:55:34 [DEBUG   ] Finished updating board flags in 1.558 seconds
2018-11-20 10:55:43 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:55:46 [DEBUG   ] Got command RPT from MCS: ref #568066
2018-11-20 10:55:46 [DEBUG   ] summary = ERROR
2018-11-20 10:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568066  13 58442 39346677 A  ERRORERROR'
2018-11-20 10:55:46 [DEBUG   ] Got command RPT from MCS: ref #568067
2018-11-20 10:55:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 10:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568067  79 58442 39346678 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 10:56:03 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 10:56:23 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:56:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:56:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:56:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:56:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:56:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:56:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 10:56:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:56:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:56:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:56:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:56:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:56:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:56:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:56:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:56:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:56:35 [DEBUG   ] Finished updating board flags in 2.210 seconds
2018-11-20 10:56:43 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:57:03 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:57:23 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:57:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:57:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:57:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:57:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:57:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:57:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:57:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:57:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 10:57:35 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:57:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:57:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 10:57:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:57:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:57:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:57:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:57:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:57:35 [DEBUG   ] Finished updating board flags in 2.077 seconds
2018-11-20 10:57:43 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:58:03 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:58:23 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:58:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:58:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:58:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:58:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:58:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:58:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 10:58:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:58:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:58:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:58:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:58:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:58:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:58:35 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 10:58:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:58:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:58:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:58:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:58:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:58:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:58:35 [DEBUG   ] Finished updating board flags in 2.023 seconds
2018-11-20 10:58:43 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 10:59:03 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 10:59:23 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 10:59:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 10:59:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 10:59:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 10:59:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 10:59:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 10:59:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 10:59:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 10:59:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 10:59:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 10:59:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 10:59:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 10:59:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 10:59:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:59:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:59:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:59:35 [WARNING ] (13, 1, 'bot')
2018-11-20 10:59:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 10:59:35 [DEBUG   ] Finished updating board flags in 1.989 seconds
2018-11-20 10:59:43 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:00:03 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:00:13 [DEBUG   ] Got command RPT from MCS: ref #568089
2018-11-20 11:00:13 [DEBUG   ] summary = ERROR
2018-11-20 11:00:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568089  13 58442 39613881 A  ERRORERROR'
2018-11-20 11:00:14 [DEBUG   ] Got command RPT from MCS: ref #568090
2018-11-20 11:00:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:00:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568090  98 58442 39614085 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:00:14 [DEBUG   ] Got command RPT from MCS: ref #568091
2018-11-20 11:00:14 [DEBUG   ] summary = ERROR
2018-11-20 11:00:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568091  13 58442 39614277 A  ERRORERROR'
2018-11-20 11:00:14 [DEBUG   ] Got command RPT from MCS: ref #568093
2018-11-20 11:00:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:00:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568093  98 58442 39614481 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:00:14 [DEBUG   ] Got command RPT from MCS: ref #568094
2018-11-20 11:00:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:00:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568094  12 58442 39614685 A  ERRORB¡ff'
2018-11-20 11:00:23 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:00:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:00:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:00:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:00:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:00:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:00:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 11:00:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:00:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:00:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:00:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:00:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:00:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:00:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:00:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 11:00:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:00:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:00:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:00:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:00:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:00:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:00:35 [DEBUG   ] Finished updating board flags in 1.813 seconds
2018-11-20 11:00:43 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:00:46 [DEBUG   ] Got command RPT from MCS: ref #568099
2018-11-20 11:00:46 [DEBUG   ] summary = ERROR
2018-11-20 11:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568099  13 58442 39646681 A  ERRORERROR'
2018-11-20 11:00:46 [DEBUG   ] Got command RPT from MCS: ref #568100
2018-11-20 11:00:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568100  98 58442 39646682 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:01:03 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:01:23 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:01:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:01:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:01:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:01:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:01:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:01:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:01:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:01:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:01:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:01:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:01:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:01:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:01:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:01:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:01:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:01:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:01:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:01:35 [DEBUG   ] Finished updating board flags in 1.848 seconds
2018-11-20 11:01:43 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:02:03 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:02:23 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:02:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:02:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:02:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:02:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:02:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:02:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:02:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 11:02:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:02:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:02:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:02:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:02:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:02:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:02:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 11:02:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:02:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:02:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:02:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:02:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:02:35 [DEBUG   ] Finished updating board flags in 1.815 seconds
2018-11-20 11:02:43 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:03:03 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:03:23 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:03:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:03:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:03:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:03:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:03:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:03:34 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:03:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 11:03:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:03:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:03:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:03:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:03:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:03:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:03:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:03:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 11:03:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 11:03:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:03:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:03:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:03:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:03:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:03:35 [DEBUG   ] Finished updating board flags in 1.830 seconds
2018-11-20 11:03:43 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:04:03 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:04:23 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 11:04:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:04:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:04:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:04:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:04:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:04:34 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:04:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:04:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:04:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:04:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:04:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:04:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:04:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:04:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:04:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:04:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:04:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:04:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:04:35 [DEBUG   ] Finished updating board flags in 1.698 seconds
2018-11-20 11:04:43 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:05:03 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:05:13 [DEBUG   ] Got command RPT from MCS: ref #568124
2018-11-20 11:05:13 [DEBUG   ] summary = ERROR
2018-11-20 11:05:13 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568124  13 58442 39913957 A  ERRORERROR'
2018-11-20 11:05:14 [DEBUG   ] Got command RPT from MCS: ref #568125
2018-11-20 11:05:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:05:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568125  98 58442 39914161 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:05:14 [DEBUG   ] Got command RPT from MCS: ref #568126
2018-11-20 11:05:14 [DEBUG   ] summary = ERROR
2018-11-20 11:05:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568126  13 58442 39914365 A  ERRORERROR'
2018-11-20 11:05:14 [DEBUG   ] Got command RPT from MCS: ref #568127
2018-11-20 11:05:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:05:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568127  98 58442 39914569 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:05:14 [DEBUG   ] Got command RPT from MCS: ref #568128
2018-11-20 11:05:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:05:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568128  12 58442 39914761 A  ERRORB¡ff'
2018-11-20 11:05:23 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:05:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:05:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371539652000000
2018-11-20 11:05:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:05:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:05:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:05:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:05:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371539652000000
2018-11-20 11:05:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:05:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:05:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:05:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:05:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:05:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:05:35 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:05:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:05:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:05:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:05:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:05:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:05:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:05:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:05:35 [DEBUG   ] Finished updating board flags in 1.624 seconds
2018-11-20 11:05:43 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:05:46 [DEBUG   ] Got command RPT from MCS: ref #568129
2018-11-20 11:05:46 [DEBUG   ] summary = ERROR
2018-11-20 11:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568129  13 58442 39946685 A  ERRORERROR'
2018-11-20 11:05:46 [DEBUG   ] Got command RPT from MCS: ref #568130
2018-11-20 11:05:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 11:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568130  71 58442 39946686 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 11:06:03 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:06:23 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:06:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:06:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371551412000000
2018-11-20 11:06:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:06:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:06:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:06:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:06:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371551412000000
2018-11-20 11:06:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:06:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:06:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:06:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:06:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:06:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:06:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:06:35 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:06:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:06:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:06:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:06:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:06:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:06:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:06:35 [DEBUG   ] Finished updating board flags in 1.624 seconds
2018-11-20 11:06:43 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:07:03 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 11:07:23 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:07:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:07:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371563172000000
2018-11-20 11:07:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:07:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:07:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:07:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:07:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371563172000000
2018-11-20 11:07:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:07:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 11:07:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:07:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:07:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:07:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:07:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:07:35 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:07:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 11:07:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:07:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:07:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:07:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:07:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:07:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:07:35 [DEBUG   ] Finished updating board flags in 1.624 seconds
2018-11-20 11:07:43 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:08:03 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:08:23 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:08:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:08:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371574932000000
2018-11-20 11:08:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:08:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:08:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:08:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:08:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371574932000000
2018-11-20 11:08:34 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:08:34 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 11:08:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:08:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:08:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:08:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:08:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:08:35 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:08:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:08:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:08:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:08:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:08:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:08:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:08:35 [DEBUG   ] Finished updating board flags in 1.590 seconds
2018-11-20 11:08:43 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:09:03 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:09:23 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:09:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:09:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371586692000000
2018-11-20 11:09:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:09:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:09:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:09:34 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:09:34 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371586692000000
2018-11-20 11:09:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:09:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:09:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371586888000000
2018-11-20 11:09:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:09:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:09:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:09:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:09:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371586888000000
2018-11-20 11:09:35 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:09:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:09:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:09:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:09:35 [WARNING ] (13, 1, 'bot')
2018-11-20 11:09:35 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:09:35 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 11:09:35 [DEBUG   ] Finished updating board flags in 1.575 seconds
2018-11-20 11:09:43 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:10:03 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:10:14 [DEBUG   ] Got command RPT from MCS: ref #568152
2018-11-20 11:10:14 [DEBUG   ] summary = ERROR
2018-11-20 11:10:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568152  13 58442 40214037 A  ERRORERROR'
2018-11-20 11:10:14 [DEBUG   ] Got command RPT from MCS: ref #568153
2018-11-20 11:10:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 11:10:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568153  79 58442 40214241 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 11:10:14 [DEBUG   ] Got command RPT from MCS: ref #568154
2018-11-20 11:10:14 [DEBUG   ] summary = ERROR
2018-11-20 11:10:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568154  13 58442 40214445 A  ERRORERROR'
2018-11-20 11:10:14 [DEBUG   ] Got command RPT from MCS: ref #568156
2018-11-20 11:10:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 11:10:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568156  79 58442 40214649 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 11:10:14 [DEBUG   ] Got command RPT from MCS: ref #568157
2018-11-20 11:10:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:10:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568157  12 58442 40214841 A  ERRORB£ff'
2018-11-20 11:10:23 [DEBUG   ] Finished updating ant stats in 0.797 seconds
2018-11-20 11:10:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:10:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:10:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:10:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:10:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:10:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 11:10:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:10:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:10:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:10:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:10:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:10:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:10:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:10:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:10:36 [DEBUG   ] Finished updating board flags in 2.267 seconds
2018-11-20 11:10:43 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:10:46 [DEBUG   ] Got command RPT from MCS: ref #568166
2018-11-20 11:10:46 [DEBUG   ] summary = ERROR
2018-11-20 11:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568166  13 58442 40246677 A  ERRORERROR'
2018-11-20 11:10:46 [DEBUG   ] Got command RPT from MCS: ref #568167
2018-11-20 11:10:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568167  98 58442 40246678 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:11:03 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 11:11:23 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 11:11:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:11:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:11:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:11:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:11:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:11:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:11:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:11:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:11:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:11:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 11:11:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:11:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:11:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:11:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:11:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:11:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:11:36 [DEBUG   ] Finished updating board flags in 2.118 seconds
2018-11-20 11:11:43 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:12:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 11:12:23 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 11:12:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:12:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:12:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:12:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:12:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:12:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:12:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:12:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:12:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:12:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:12:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:12:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:12:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:12:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:12:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:12:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:12:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:12:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:12:36 [DEBUG   ] Finished updating board flags in 2.018 seconds
2018-11-20 11:12:43 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 11:13:03 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:13:23 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 11:13:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:13:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:13:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:13:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:13:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:13:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 11:13:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 11:13:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:13:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:13:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:13:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:13:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:13:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:13:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:13:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 5
2018-11-20 11:13:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:13:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:13:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:13:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:13:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:13:36 [DEBUG   ] Finished updating board flags in 2.037 seconds
2018-11-20 11:13:43 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:14:03 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 11:14:23 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:14:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:14:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:14:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:14:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:14:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:14:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:14:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 11:14:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:14:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:14:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:14:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:14:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:14:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:14:36 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:14:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:14:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:14:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:14:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:14:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:14:36 [DEBUG   ] Finished updating board flags in 2.013 seconds
2018-11-20 11:14:43 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:15:03 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:15:14 [DEBUG   ] Got command RPT from MCS: ref #568187
2018-11-20 11:15:14 [DEBUG   ] summary = ERROR
2018-11-20 11:15:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568187  13 58442 40514125 A  ERRORERROR'
2018-11-20 11:15:14 [DEBUG   ] Got command RPT from MCS: ref #568188
2018-11-20 11:15:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:15:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568188  98 58442 40514317 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:15:14 [DEBUG   ] Got command RPT from MCS: ref #568189
2018-11-20 11:15:14 [DEBUG   ] summary = ERROR
2018-11-20 11:15:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568189  13 58442 40514521 A  ERRORERROR'
2018-11-20 11:15:14 [DEBUG   ] Got command RPT from MCS: ref #568190
2018-11-20 11:15:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:15:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568190  98 58442 40514725 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:15:14 [DEBUG   ] Got command RPT from MCS: ref #568191
2018-11-20 11:15:14 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:15:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568191  12 58442 40514929 A  ERRORB£ff'
2018-11-20 11:15:23 [DEBUG   ] Finished updating ant stats in 0.598 seconds
2018-11-20 11:15:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:15:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:15:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:15:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:15:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:15:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 11:15:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 3
2018-11-20 11:15:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:15:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:15:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:15:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:15:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:15:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:15:36 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:15:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:15:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:15:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:15:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:15:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:15:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:15:36 [DEBUG   ] Finished updating board flags in 1.945 seconds
2018-11-20 11:15:43 [DEBUG   ] Finished updating ant stats in 0.401 seconds
2018-11-20 11:15:46 [DEBUG   ] Got command RPT from MCS: ref #568192
2018-11-20 11:15:46 [DEBUG   ] summary = ERROR
2018-11-20 11:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568192  13 58442 40546689 A  ERRORERROR'
2018-11-20 11:15:46 [DEBUG   ] Got command RPT from MCS: ref #568193
2018-11-20 11:15:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568193  98 58442 40546690 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:16:04 [DEBUG   ] Finished updating ant stats in 1.399 seconds
2018-11-20 11:16:24 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 11:16:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:16:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:16:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:16:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:16:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:16:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:16:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:16:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:16:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:16:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:16:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:16:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:16:36 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:16:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:16:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:16:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:16:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:16:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:16:36 [DEBUG   ] Finished updating board flags in 1.818 seconds
2018-11-20 11:16:44 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 11:17:04 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 11:17:24 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 11:17:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:17:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:17:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:17:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:17:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:17:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:17:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:17:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:17:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:17:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:17:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:17:36 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:17:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:17:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:17:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:17:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:17:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:17:36 [DEBUG   ] Finished updating board flags in 1.788 seconds
2018-11-20 11:17:44 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 11:18:04 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 11:18:24 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 11:18:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:18:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371692728000000
2018-11-20 11:18:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:18:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:18:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:18:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:18:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371692728000000
2018-11-20 11:18:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:18:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:18:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:18:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:18:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:18:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:18:36 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:18:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:18:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:18:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:18:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:18:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:18:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:18:36 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:18:36 [DEBUG   ] Finished updating board flags in 1.661 seconds
2018-11-20 11:18:44 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 11:19:04 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:19:24 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:19:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:19:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371704488000000
2018-11-20 11:19:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:19:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:19:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:19:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:19:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371704488000000
2018-11-20 11:19:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 11:19:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 11:19:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:19:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:19:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:19:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:19:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:19:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:19:36 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:19:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:19:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:19:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:19:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:19:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:19:36 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:19:36 [DEBUG   ] Finished updating board flags in 1.524 seconds
2018-11-20 11:19:44 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 11:20:04 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 11:20:14 [DEBUG   ] Got command RPT from MCS: ref #568219
2018-11-20 11:20:14 [DEBUG   ] summary = ERROR
2018-11-20 11:20:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568219  13 58442 40814201 A  ERRORERROR'
2018-11-20 11:20:14 [DEBUG   ] Got command RPT from MCS: ref #568220
2018-11-20 11:20:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 11:20:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568220  71 58442 40814405 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 11:20:14 [DEBUG   ] Got command RPT from MCS: ref #568221
2018-11-20 11:20:14 [DEBUG   ] summary = ERROR
2018-11-20 11:20:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568221  13 58442 40814609 A  ERRORERROR'
2018-11-20 11:20:14 [DEBUG   ] Got command RPT from MCS: ref #568223
2018-11-20 11:20:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 11:20:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568223  71 58442 40814801 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 11:20:15 [DEBUG   ] Got command RPT from MCS: ref #568224
2018-11-20 11:20:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:20:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568224  12 58442 40815005 A  ERRORB¢ff'
2018-11-20 11:20:24 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 11:20:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:20:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371716248000000
2018-11-20 11:20:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:20:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:20:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:20:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:20:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371716248000000
2018-11-20 11:20:35 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 11:20:35 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:20:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:20:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:20:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:20:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:20:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:20:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 11:20:36 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:20:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:20:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:20:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:20:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:20:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:20:36 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:20:36 [DEBUG   ] Finished updating board flags in 1.465 seconds
2018-11-20 11:20:44 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:20:46 [DEBUG   ] Got command RPT from MCS: ref #568229
2018-11-20 11:20:46 [DEBUG   ] summary = ERROR
2018-11-20 11:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568229  13 58442 40846685 A  ERRORERROR'
2018-11-20 11:20:46 [DEBUG   ] Got command RPT from MCS: ref #568230
2018-11-20 11:20:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 11:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568230  71 58442 40846686 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 11:21:04 [DEBUG   ] Finished updating ant stats in 0.999 seconds
2018-11-20 11:21:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:21:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:21:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371728008000000
2018-11-20 11:21:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:21:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:21:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:21:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:21:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371728008000000
2018-11-20 11:21:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:21:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:21:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371728204000000
2018-11-20 11:21:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:21:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:21:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:21:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:21:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371728204000000
2018-11-20 11:21:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:21:36 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:21:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:21:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:21:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:21:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:21:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:21:36 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 11:21:36 [DEBUG   ] Finished updating board flags in 1.472 seconds
2018-11-20 11:21:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:22:04 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:22:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:22:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:22:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371739768000000
2018-11-20 11:22:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:22:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:22:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:22:35 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:22:35 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371739768000000
2018-11-20 11:22:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:22:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:22:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371739964000000
2018-11-20 11:22:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:22:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:22:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:22:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:22:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371739964000000
2018-11-20 11:22:36 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:22:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 11:22:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:22:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:22:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:22:36 [WARNING ] (13, 1, 'bot')
2018-11-20 11:22:36 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:22:36 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 11:22:36 [DEBUG   ] Finished updating board flags in 1.518 seconds
2018-11-20 11:22:44 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:23:04 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:23:24 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:23:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:23:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:23:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:23:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:23:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:23:36 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:23:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:23:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:23:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:23:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:23:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:23:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:23:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:23:37 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:23:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 11:23:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:23:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:23:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:23:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:23:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:23:37 [DEBUG   ] Finished updating board flags in 2.211 seconds
2018-11-20 11:23:44 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:24:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:24:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:24:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:24:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:24:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:24:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:24:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:24:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 11:24:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 11:24:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:24:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:24:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:24:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:24:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:24:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:24:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:24:37 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:24:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:24:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:24:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:24:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:24:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:24:37 [DEBUG   ] Finished updating board flags in 2.189 seconds
2018-11-20 11:24:44 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:25:04 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:25:14 [DEBUG   ] Got command RPT from MCS: ref #568250
2018-11-20 11:25:14 [DEBUG   ] summary = ERROR
2018-11-20 11:25:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568250  13 58442 41114285 A  ERRORERROR'
2018-11-20 11:25:14 [DEBUG   ] Got command RPT from MCS: ref #568251
2018-11-20 11:25:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:25:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568251  98 58442 41114477 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:25:14 [DEBUG   ] Got command RPT from MCS: ref #568252
2018-11-20 11:25:14 [DEBUG   ] summary = ERROR
2018-11-20 11:25:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568252  13 58442 41114681 A  ERRORERROR'
2018-11-20 11:25:14 [DEBUG   ] Got command RPT from MCS: ref #568253
2018-11-20 11:25:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:25:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568253  98 58442 41114885 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:25:15 [DEBUG   ] Got command RPT from MCS: ref #568254
2018-11-20 11:25:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:25:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568254  12 58442 41115089 A  ERRORB¡ff'
2018-11-20 11:25:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:25:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:25:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:25:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:25:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:25:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:25:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:25:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:25:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:25:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:25:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:25:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:25:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:25:37 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:25:37 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 11:25:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:25:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:25:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:25:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:25:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:25:37 [DEBUG   ] Finished updating board flags in 1.992 seconds
2018-11-20 11:25:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:25:46 [DEBUG   ] Got command RPT from MCS: ref #568259
2018-11-20 11:25:46 [DEBUG   ] summary = ERROR
2018-11-20 11:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568259  13 58442 41146689 A  ERRORERROR'
2018-11-20 11:25:46 [DEBUG   ] Got command RPT from MCS: ref #568260
2018-11-20 11:25:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568260  98 58442 41146690 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:26:04 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:26:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:26:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:26:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:26:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:26:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:26:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:26:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:26:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:26:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:26:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:26:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:26:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:26:37 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:26:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:26:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:26:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:26:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:26:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:26:37 [DEBUG   ] Finished updating board flags in 1.821 seconds
2018-11-20 11:26:44 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:27:04 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:27:24 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:27:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:27:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:27:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:27:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:27:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:27:36 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:27:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:27:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:27:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:27:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:27:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:27:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:27:37 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:27:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:27:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:27:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:27:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:27:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:27:37 [DEBUG   ] Finished updating board flags in 1.758 seconds
2018-11-20 11:27:44 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:28:04 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:28:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:28:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:28:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371810524000000
2018-11-20 11:28:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:28:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:28:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:28:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:28:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371810524000000
2018-11-20 11:28:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:28:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:28:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:28:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:28:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:28:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:28:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:28:37 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:28:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:28:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:28:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:28:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:28:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:28:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:28:37 [DEBUG   ] Finished updating board flags in 1.656 seconds
2018-11-20 11:28:44 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 11:29:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:29:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:29:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:29:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371822284000000
2018-11-20 11:29:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:29:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:29:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:29:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:29:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371822284000000
2018-11-20 11:29:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:29:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:29:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:29:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:29:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:29:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:29:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 11:29:37 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:29:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:29:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:29:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:29:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:29:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:29:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:29:37 [DEBUG   ] Finished updating board flags in 1.541 seconds
2018-11-20 11:29:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:30:04 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:30:14 [DEBUG   ] Got command RPT from MCS: ref #568282
2018-11-20 11:30:14 [DEBUG   ] summary = ERROR
2018-11-20 11:30:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568282  13 58442 41414361 A  ERRORERROR'
2018-11-20 11:30:14 [DEBUG   ] Got command RPT from MCS: ref #568283
2018-11-20 11:30:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 11:30:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568283  71 58442 41414565 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 11:30:14 [DEBUG   ] Got command RPT from MCS: ref #568284
2018-11-20 11:30:14 [DEBUG   ] summary = ERROR
2018-11-20 11:30:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568284  13 58442 41414757 A  ERRORERROR'
2018-11-20 11:30:14 [DEBUG   ] Got command RPT from MCS: ref #568286
2018-11-20 11:30:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 11:30:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568286  71 58442 41414961 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 11:30:15 [DEBUG   ] Got command RPT from MCS: ref #568288
2018-11-20 11:30:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:30:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568288  12 58442 41415165 A  ERRORB ff'
2018-11-20 11:30:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:30:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:30:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371834044000000
2018-11-20 11:30:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:30:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:30:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:30:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:30:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371834044000000
2018-11-20 11:30:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 11:30:36 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 11:30:36 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:30:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:30:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371834240000000
2018-11-20 11:30:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:30:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:30:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:30:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:30:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371834240000000
2018-11-20 11:30:37 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:30:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:30:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:30:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:30:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:30:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:30:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 11:30:37 [DEBUG   ] Finished updating board flags in 1.522 seconds
2018-11-20 11:30:44 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:30:46 [DEBUG   ] Got command RPT from MCS: ref #568292
2018-11-20 11:30:46 [DEBUG   ] summary = ERROR
2018-11-20 11:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568292  13 58442 41446693 A  ERRORERROR'
2018-11-20 11:30:46 [DEBUG   ] Got command RPT from MCS: ref #568293
2018-11-20 11:30:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 11:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568293  79 58442 41446694 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 11:31:04 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:31:24 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:31:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:31:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371845804000000
2018-11-20 11:31:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:31:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:31:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:31:36 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:31:36 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371845804000000
2018-11-20 11:31:36 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:31:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 11:31:36 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 11:31:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:31:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:31:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371846000000000
2018-11-20 11:31:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:31:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:31:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:31:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:31:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371846000000000
2018-11-20 11:31:37 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:31:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:31:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:31:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:31:37 [WARNING ] (13, 1, 'bot')
2018-11-20 11:31:37 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:31:37 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 11:31:37 [DEBUG   ] Finished updating board flags in 1.565 seconds
2018-11-20 11:31:44 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:32:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:32:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:32:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:32:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:32:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:32:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:32:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:32:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:32:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:32:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:32:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 11:32:38 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:32:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:32:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:32:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:32:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:32:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:32:38 [DEBUG   ] Finished updating board flags in 2.191 seconds
2018-11-20 11:32:44 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:33:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:33:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:33:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:33:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:33:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:33:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:33:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:33:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:33:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:33:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:33:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:33:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:33:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:33:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:33:38 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:33:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:33:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:33:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:33:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:33:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:33:38 [DEBUG   ] Finished updating board flags in 2.049 seconds
2018-11-20 11:33:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:34:04 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:34:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:34:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:34:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:34:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:34:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:34:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:34:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:34:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:34:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 11:34:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:34:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:34:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:34:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:34:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:34:38 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:34:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:34:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:34:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:34:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:34:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:34:38 [DEBUG   ] Finished updating board flags in 1.956 seconds
2018-11-20 11:34:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:35:04 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:35:14 [DEBUG   ] Got command RPT from MCS: ref #568317
2018-11-20 11:35:14 [DEBUG   ] summary = ERROR
2018-11-20 11:35:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568317  13 58442 41714437 A  ERRORERROR'
2018-11-20 11:35:14 [DEBUG   ] Got command RPT from MCS: ref #568318
2018-11-20 11:35:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:35:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568318  98 58442 41714641 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:35:14 [DEBUG   ] Got command RPT from MCS: ref #568319
2018-11-20 11:35:14 [DEBUG   ] summary = ERROR
2018-11-20 11:35:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568319  13 58442 41714845 A  ERRORERROR'
2018-11-20 11:35:15 [DEBUG   ] Got command RPT from MCS: ref #568320
2018-11-20 11:35:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:35:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568320  98 58442 41715049 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:35:15 [DEBUG   ] Got command RPT from MCS: ref #568321
2018-11-20 11:35:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:35:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568321  12 58442 41715241 A  ERRORB¡ff'
2018-11-20 11:35:24 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:35:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:35:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:35:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:35:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:35:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:35:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:35:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:35:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:35:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:35:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:35:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:35:38 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:35:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:35:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:35:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:35:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:35:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:35:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:35:38 [DEBUG   ] Finished updating board flags in 1.734 seconds
2018-11-20 11:35:44 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:35:46 [DEBUG   ] Got command RPT from MCS: ref #568322
2018-11-20 11:35:46 [DEBUG   ] summary = ERROR
2018-11-20 11:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568322  13 58442 41746685 A  ERRORERROR'
2018-11-20 11:35:46 [DEBUG   ] Got command RPT from MCS: ref #568323
2018-11-20 11:35:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568323  98 58442 41746686 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:36:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:36:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:36:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:36:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371904800000000
2018-11-20 11:36:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:36:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:36:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:36:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:36:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371904800000000
2018-11-20 11:36:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 11:36:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:36:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:36:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:36:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:36:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:36:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:36:38 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:36:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:36:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:36:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:36:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:36:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:36:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:36:38 [DEBUG   ] Finished updating board flags in 1.623 seconds
2018-11-20 11:36:44 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:37:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:37:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:37:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:37:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371916560000000
2018-11-20 11:37:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:37:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:37:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:37:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:37:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371916560000000
2018-11-20 11:37:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:37:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 11:37:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:37:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:37:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:37:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:37:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:37:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:37:38 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:37:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:37:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:37:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:37:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:37:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:37:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:37:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:37:38 [DEBUG   ] Finished updating board flags in 1.576 seconds
2018-11-20 11:37:44 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:38:04 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:38:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:38:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:38:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371928320000000
2018-11-20 11:38:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:38:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:38:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:38:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:38:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371928320000000
2018-11-20 11:38:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 11:38:37 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:38:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:38:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371928516000000
2018-11-20 11:38:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:38:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:38:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:38:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:38:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371928516000000
2018-11-20 11:38:38 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:38:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:38:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:38:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:38:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:38:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:38:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 11:38:38 [DEBUG   ] Finished updating board flags in 1.556 seconds
2018-11-20 11:38:44 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:39:04 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:39:24 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:39:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:39:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302371940080000000
2018-11-20 11:39:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:39:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:39:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:39:37 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:39:37 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302371940080000000
2018-11-20 11:39:37 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:39:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:39:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:39:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302371940276000000
2018-11-20 11:39:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:39:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:39:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:39:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:39:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302371940276000000
2018-11-20 11:39:38 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 11:39:38 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:39:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:39:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:39:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:39:38 [WARNING ] (13, 1, 'bot')
2018-11-20 11:39:38 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:39:38 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 11:39:38 [DEBUG   ] Finished updating board flags in 1.580 seconds
2018-11-20 11:39:44 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:40:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:40:14 [DEBUG   ] Got command RPT from MCS: ref #568345
2018-11-20 11:40:14 [DEBUG   ] summary = ERROR
2018-11-20 11:40:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568345  13 58442 42014521 A  ERRORERROR'
2018-11-20 11:40:14 [DEBUG   ] Got command RPT from MCS: ref #568346
2018-11-20 11:40:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 11:40:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568346  79 58442 42014725 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 11:40:14 [DEBUG   ] Got command RPT from MCS: ref #568347
2018-11-20 11:40:14 [DEBUG   ] summary = ERROR
2018-11-20 11:40:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568347  13 58442 42014921 A  ERRORERROR'
2018-11-20 11:40:15 [DEBUG   ] Got command RPT from MCS: ref #568349
2018-11-20 11:40:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 11:40:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568349  79 58442 42015125 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 11:40:15 [DEBUG   ] Got command RPT from MCS: ref #568350
2018-11-20 11:40:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:40:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568350  12 58442 42015329 A  ERRORB£ff'
2018-11-20 11:40:24 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:40:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:40:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:40:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:40:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:40:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:40:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:40:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:40:39 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:40:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:40:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:40:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:40:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:40:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:40:39 [DEBUG   ] Finished updating board flags in 2.201 seconds
2018-11-20 11:40:44 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:40:46 [DEBUG   ] Got command RPT from MCS: ref #568359
2018-11-20 11:40:46 [DEBUG   ] summary = ERROR
2018-11-20 11:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568359  13 58442 42046689 A  ERRORERROR'
2018-11-20 11:40:46 [DEBUG   ] Got command RPT from MCS: ref #568360
2018-11-20 11:40:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568360  98 58442 42046690 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:41:04 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:41:24 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:41:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:41:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:41:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:41:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:41:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:41:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:41:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:41:39 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:41:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:41:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:41:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:41:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:41:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:41:39 [DEBUG   ] Finished updating board flags in 2.077 seconds
2018-11-20 11:41:44 [DEBUG   ] Finished updating ant stats in 0.798 seconds
2018-11-20 11:42:04 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:42:24 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:42:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:42:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:42:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:42:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:42:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:42:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:42:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:42:39 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:42:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:42:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:42:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:42:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:42:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:42:39 [DEBUG   ] Finished updating board flags in 2.020 seconds
2018-11-20 11:42:44 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:43:04 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:43:24 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:43:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:43:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:43:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:43:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:43:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:43:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:43:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 11:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:43:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:43:39 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:43:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:43:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:43:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:43:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:43:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:43:39 [DEBUG   ] Finished updating board flags in 1.976 seconds
2018-11-20 11:43:44 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:44:04 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:44:24 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 11:44:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:44:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:44:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:44:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:44:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:44:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:44:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 11:44:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:44:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:44:39 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:44:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:44:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:44:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:44:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:44:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:44:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:44:39 [DEBUG   ] Finished updating board flags in 1.825 seconds
2018-11-20 11:44:44 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:45:04 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:45:14 [DEBUG   ] Got command RPT from MCS: ref #568380
2018-11-20 11:45:14 [DEBUG   ] summary = ERROR
2018-11-20 11:45:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568380  13 58442 42314605 A  ERRORERROR'
2018-11-20 11:45:14 [DEBUG   ] Got command RPT from MCS: ref #568381
2018-11-20 11:45:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:45:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568381  98 58442 42314797 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:45:15 [DEBUG   ] Got command RPT from MCS: ref #568382
2018-11-20 11:45:15 [DEBUG   ] summary = ERROR
2018-11-20 11:45:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568382  13 58442 42315001 A  ERRORERROR'
2018-11-20 11:45:15 [DEBUG   ] Got command RPT from MCS: ref #568383
2018-11-20 11:45:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:45:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568383  98 58442 42315205 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:45:15 [DEBUG   ] Got command RPT from MCS: ref #568384
2018-11-20 11:45:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:45:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568384  12 58442 42315409 A  ERRORB¢ff'
2018-11-20 11:45:24 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:45:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:45:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:45:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:45:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:45:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:45:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 11:45:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:45:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:45:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:45:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:45:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:45:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:45:39 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:45:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 11:45:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:45:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:45:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:45:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:45:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:45:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:45:39 [DEBUG   ] Finished updating board flags in 1.806 seconds
2018-11-20 11:45:44 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 11:45:46 [DEBUG   ] Got command RPT from MCS: ref #568385
2018-11-20 11:45:46 [DEBUG   ] summary = ERROR
2018-11-20 11:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568385  13 58442 42346689 A  ERRORERROR'
2018-11-20 11:45:46 [DEBUG   ] Got command RPT from MCS: ref #568386
2018-11-20 11:45:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568386  98 58442 42346690 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:46:04 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:46:24 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:46:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:46:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:46:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:46:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:46:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:46:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:46:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:46:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:46:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:46:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:46:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:46:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:46:39 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:46:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 11:46:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:46:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:46:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:46:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:46:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:46:39 [DEBUG   ] Finished updating board flags in 1.910 seconds
2018-11-20 11:46:44 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:47:04 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 11:47:24 [DEBUG   ] Finished updating ant stats in 0.600 seconds
2018-11-20 11:47:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:47:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:47:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:47:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:47:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:47:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:47:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:47:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:47:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:47:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:47:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:47:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 11:47:39 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:47:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:47:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:47:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:47:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:47:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:47:39 [DEBUG   ] Finished updating board flags in 1.676 seconds
2018-11-20 11:47:44 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 11:48:04 [DEBUG   ] Finished updating ant stats in 0.591 seconds
2018-11-20 11:48:24 [DEBUG   ] Finished updating ant stats in 0.402 seconds
2018-11-20 11:48:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:48:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372046116000000
2018-11-20 11:48:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:48:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:48:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:48:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:48:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372046116000000
2018-11-20 11:48:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 11:48:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:48:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:48:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:48:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:48:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:48:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:48:39 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:48:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:48:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:48:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:48:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:48:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:48:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:48:39 [DEBUG   ] Finished updating board flags in 1.610 seconds
2018-11-20 11:48:45 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-20 11:49:05 [DEBUG   ] Finished updating ant stats in 1.206 seconds
2018-11-20 11:49:25 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 11:49:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:49:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372057876000000
2018-11-20 11:49:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:49:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:49:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:49:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:49:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372057876000000
2018-11-20 11:49:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:49:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:49:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:49:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:49:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:49:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:49:39 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:49:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 11:49:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:49:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:49:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:49:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:49:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:49:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:49:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:49:39 [DEBUG   ] Finished updating board flags in 1.595 seconds
2018-11-20 11:49:45 [DEBUG   ] Finished updating ant stats in 1.197 seconds
2018-11-20 11:50:05 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:50:14 [DEBUG   ] Got command RPT from MCS: ref #568412
2018-11-20 11:50:14 [DEBUG   ] summary = ERROR
2018-11-20 11:50:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568412  13 58442 42614677 A  ERRORERROR'
2018-11-20 11:50:14 [DEBUG   ] Got command RPT from MCS: ref #568413
2018-11-20 11:50:14 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 11:50:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568413  71 58442 42614882 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 11:50:15 [DEBUG   ] Got command RPT from MCS: ref #568414
2018-11-20 11:50:15 [DEBUG   ] summary = ERROR
2018-11-20 11:50:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568414  13 58442 42615085 A  ERRORERROR'
2018-11-20 11:50:15 [DEBUG   ] Got command RPT from MCS: ref #568416
2018-11-20 11:50:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 11:50:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568416  71 58442 42615277 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 11:50:15 [DEBUG   ] Got command RPT from MCS: ref #568418
2018-11-20 11:50:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:50:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568418  12 58442 42615481 A  ERRORB¡ff'
2018-11-20 11:50:25 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 11:50:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:50:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372069636000000
2018-11-20 11:50:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:50:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:50:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:50:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:50:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372069636000000
2018-11-20 11:50:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:50:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:50:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:50:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:50:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:50:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:50:39 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:50:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 11:50:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:50:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:50:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:50:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:50:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:50:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:50:39 [DEBUG   ] Finished updating board flags in 1.418 seconds
2018-11-20 11:50:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 11:50:46 [DEBUG   ] Got command RPT from MCS: ref #568422
2018-11-20 11:50:46 [DEBUG   ] summary = ERROR
2018-11-20 11:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568422  13 58442 42646689 A  ERRORERROR'
2018-11-20 11:50:46 [DEBUG   ] Got command RPT from MCS: ref #568423
2018-11-20 11:50:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 11:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568423  71 58442 42646690 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 11:51:05 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:51:25 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 11:51:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:51:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372081396000000
2018-11-20 11:51:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:51:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:51:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:51:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:51:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372081396000000
2018-11-20 11:51:38 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:51:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:51:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:51:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:51:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:51:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:51:39 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:51:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 11:51:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:51:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:51:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:51:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:51:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:51:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 11:51:39 [DEBUG   ] Finished updating board flags in 1.381 seconds
2018-11-20 11:51:45 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 11:52:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 11:52:25 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:52:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:52:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372093156000000
2018-11-20 11:52:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:52:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:52:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:52:38 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:52:38 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372093156000000
2018-11-20 11:52:38 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 11:52:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:52:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:52:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372093352000000
2018-11-20 11:52:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:52:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:52:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:52:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:52:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372093352000000
2018-11-20 11:52:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:52:39 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:52:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:52:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:52:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:52:39 [WARNING ] (13, 1, 'bot')
2018-11-20 11:52:39 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:52:39 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 11:52:39 [DEBUG   ] Finished updating board flags in 1.373 seconds
2018-11-20 11:52:45 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:53:05 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 11:53:25 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 11:53:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:53:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:53:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:53:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:53:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:53:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:53:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:53:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:53:40 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:53:40 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:53:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:53:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:53:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:53:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:53:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:53:40 [DEBUG   ] Finished updating board flags in 2.047 seconds
2018-11-20 11:53:45 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:54:05 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 11:54:25 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 11:54:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:54:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:54:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:54:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:54:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:54:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 11:54:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:54:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:54:40 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:54:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:54:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:54:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:54:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:54:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:54:40 [DEBUG   ] Finished updating board flags in 2.096 seconds
2018-11-20 11:54:45 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 11:55:05 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:55:14 [DEBUG   ] Got command RPT from MCS: ref #568443
2018-11-20 11:55:14 [DEBUG   ] summary = ERROR
2018-11-20 11:55:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568443  13 58442 42914761 A  ERRORERROR'
2018-11-20 11:55:14 [DEBUG   ] Got command RPT from MCS: ref #568444
2018-11-20 11:55:14 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:55:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568444  98 58442 42914965 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:55:15 [DEBUG   ] Got command RPT from MCS: ref #568445
2018-11-20 11:55:15 [DEBUG   ] summary = ERROR
2018-11-20 11:55:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568445  13 58442 42915157 A  ERRORERROR'
2018-11-20 11:55:15 [DEBUG   ] Got command RPT from MCS: ref #568446
2018-11-20 11:55:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:55:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568446  98 58442 42915361 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:55:15 [DEBUG   ] Got command RPT from MCS: ref #568447
2018-11-20 11:55:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 11:55:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568447  12 58442 42915565 A  ERRORB ff'
2018-11-20 11:55:25 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:55:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:55:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:55:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:55:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:55:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:55:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:55:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:55:40 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:55:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:55:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:55:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:55:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:55:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:55:40 [DEBUG   ] Finished updating board flags in 2.023 seconds
2018-11-20 11:55:45 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 11:55:46 [DEBUG   ] Got command RPT from MCS: ref #568452
2018-11-20 11:55:46 [DEBUG   ] summary = ERROR
2018-11-20 11:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568452  13 58442 42946689 A  ERRORERROR'
2018-11-20 11:55:46 [DEBUG   ] Got command RPT from MCS: ref #568453
2018-11-20 11:55:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 11:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568453  98 58442 42946690 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 11:56:05 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 11:56:25 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:56:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:56:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:56:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:56:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:56:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:56:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:56:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 11:56:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:56:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:56:40 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:56:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:56:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:56:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:56:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:56:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:56:40 [DEBUG   ] Finished updating board flags in 1.869 seconds
2018-11-20 11:56:45 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 11:57:05 [DEBUG   ] Finished updating ant stats in 0.991 seconds
2018-11-20 11:57:25 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:57:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:57:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:57:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:57:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:57:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:57:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 11:57:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:57:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:57:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 11:57:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 11:57:40 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 11:57:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:57:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:57:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:57:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:57:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:57:40 [DEBUG   ] Finished updating board flags in 1.812 seconds
2018-11-20 11:57:45 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:58:05 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:58:25 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 11:58:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:58:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:58:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:58:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:58:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:58:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:58:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 11:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:58:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:58:40 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:58:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 11:58:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:58:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:58:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:58:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:58:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:58:40 [DEBUG   ] Finished updating board flags in 1.814 seconds
2018-11-20 11:58:45 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 11:59:05 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:59:25 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 11:59:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 11:59:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 11:59:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 11:59:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 11:59:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 11:59:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 11:59:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 11:59:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 11:59:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 11:59:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 11:59:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 11:59:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 11:59:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 11:59:40 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 11:59:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:59:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:59:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:59:40 [WARNING ] (13, 1, 'bot')
2018-11-20 11:59:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 11:59:40 [DEBUG   ] Finished updating board flags in 1.788 seconds
2018-11-20 11:59:45 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:00:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:00:14 [DEBUG   ] Got command RPT from MCS: ref #568475
2018-11-20 12:00:14 [DEBUG   ] summary = ERROR
2018-11-20 12:00:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568475  13 58442 43214841 A  ERRORERROR'
2018-11-20 12:00:15 [DEBUG   ] Got command RPT from MCS: ref #568476
2018-11-20 12:00:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:00:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568476  98 58442 43215045 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:00:15 [DEBUG   ] Got command RPT from MCS: ref #568477
2018-11-20 12:00:15 [DEBUG   ] summary = ERROR
2018-11-20 12:00:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568477  13 58442 43215237 A  ERRORERROR'
2018-11-20 12:00:15 [DEBUG   ] Got command RPT from MCS: ref #568479
2018-11-20 12:00:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:00:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568479  98 58442 43215441 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:00:15 [DEBUG   ] Got command RPT from MCS: ref #568480
2018-11-20 12:00:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:00:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568480  12 58442 43215645 A  ERRORB¡ff'
2018-11-20 12:00:25 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:00:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:00:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372187432000000
2018-11-20 12:00:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:00:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:00:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:00:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:00:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372187432000000
2018-11-20 12:00:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:00:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:00:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:00:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:00:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:00:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:00:40 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:00:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:00:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:00:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:00:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:00:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:00:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:00:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:00:40 [DEBUG   ] Finished updating board flags in 1.627 seconds
2018-11-20 12:00:45 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:00:46 [DEBUG   ] Got command RPT from MCS: ref #568485
2018-11-20 12:00:46 [DEBUG   ] summary = ERROR
2018-11-20 12:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568485  13 58442 43246693 A  ERRORERROR'
2018-11-20 12:00:46 [DEBUG   ] Got command RPT from MCS: ref #568486
2018-11-20 12:00:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568486  71 58442 43246694 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:01:05 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:01:25 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:01:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:01:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372199192000000
2018-11-20 12:01:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:01:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:01:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:01:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:01:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372199192000000
2018-11-20 12:01:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:01:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:01:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:01:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:01:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:01:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:01:40 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:01:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:01:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:01:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:01:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:01:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:01:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:01:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:01:40 [DEBUG   ] Finished updating board flags in 1.667 seconds
2018-11-20 12:01:45 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:02:05 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:02:25 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:02:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:02:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372210952000000
2018-11-20 12:02:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:02:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:02:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:02:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:02:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372210952000000
2018-11-20 12:02:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:02:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:02:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:02:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:02:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:02:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:02:40 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:02:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:02:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:02:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:02:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:02:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:02:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:02:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:02:40 [DEBUG   ] Finished updating board flags in 1.634 seconds
2018-11-20 12:02:45 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:03:05 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:03:25 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:03:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:03:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372222712000000
2018-11-20 12:03:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:03:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:03:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:03:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:03:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372222712000000
2018-11-20 12:03:39 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:03:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:03:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:03:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:03:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:03:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:03:40 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:03:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:03:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:03:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:03:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:03:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:03:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:03:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:03:40 [DEBUG   ] Finished updating board flags in 1.493 seconds
2018-11-20 12:03:45 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:04:05 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:04:25 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:04:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:04:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372234472000000
2018-11-20 12:04:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:04:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:04:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:04:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:04:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372234472000000
2018-11-20 12:04:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 12:04:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:04:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 12:04:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:04:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372234668000000
2018-11-20 12:04:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:04:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:04:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:04:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:04:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372234668000000
2018-11-20 12:04:40 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:04:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:04:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:04:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:04:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:04:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:04:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:04:40 [DEBUG   ] Finished updating board flags in 1.499 seconds
2018-11-20 12:04:45 [DEBUG   ] Finished updating ant stats in 0.799 seconds
2018-11-20 12:05:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:05:14 [DEBUG   ] Got command RPT from MCS: ref #568510
2018-11-20 12:05:14 [DEBUG   ] summary = ERROR
2018-11-20 12:05:14 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568510  13 58442 43514925 A  ERRORERROR'
2018-11-20 12:05:15 [DEBUG   ] Got command RPT from MCS: ref #568511
2018-11-20 12:05:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 12:05:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568511  79 58442 43515117 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 12:05:15 [DEBUG   ] Got command RPT from MCS: ref #568512
2018-11-20 12:05:15 [DEBUG   ] summary = ERROR
2018-11-20 12:05:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568512  13 58442 43515321 A  ERRORERROR'
2018-11-20 12:05:15 [DEBUG   ] Got command RPT from MCS: ref #568513
2018-11-20 12:05:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 12:05:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568513  79 58442 43515525 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 12:05:15 [DEBUG   ] Got command RPT from MCS: ref #568514
2018-11-20 12:05:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:05:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568514  12 58442 43515729 A  ERRORB¢ff'
2018-11-20 12:05:25 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:05:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:05:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372246232000000
2018-11-20 12:05:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:05:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:05:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:05:39 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:05:39 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372246232000000
2018-11-20 12:05:39 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 12:05:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:05:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:05:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372246428000000
2018-11-20 12:05:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:05:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:05:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:05:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:05:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372246428000000
2018-11-20 12:05:40 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:05:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:05:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:05:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:05:40 [WARNING ] (13, 1, 'bot')
2018-11-20 12:05:40 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:05:40 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:05:40 [DEBUG   ] Finished updating board flags in 1.501 seconds
2018-11-20 12:05:45 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:05:46 [DEBUG   ] Got command RPT from MCS: ref #568515
2018-11-20 12:05:46 [DEBUG   ] summary = ERROR
2018-11-20 12:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568515  13 58442 43546693 A  ERRORERROR'
2018-11-20 12:05:46 [DEBUG   ] Got command RPT from MCS: ref #568516
2018-11-20 12:05:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 12:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568516  79 58442 43546694 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 12:06:05 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:06:25 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:06:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:06:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:06:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:06:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:06:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:06:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 12:06:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:06:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:06:41 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:06:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:06:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:06:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:06:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:06:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:06:41 [DEBUG   ] Finished updating board flags in 2.215 seconds
2018-11-20 12:06:45 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:07:05 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:07:25 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:07:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:07:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:07:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:07:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:07:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:07:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:07:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:07:41 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:07:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:07:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:07:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:07:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:07:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:07:41 [DEBUG   ] Finished updating board flags in 2.227 seconds
2018-11-20 12:07:45 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:08:05 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:08:25 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:08:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:08:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:08:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:08:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:08:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:08:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:08:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:08:41 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:08:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 12:08:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:08:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:08:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:08:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:08:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:08:41 [DEBUG   ] Finished updating board flags in 2.221 seconds
2018-11-20 12:08:45 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:09:05 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 12:09:25 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:09:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:09:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:09:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:09:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:09:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:09:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 12:09:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:09:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:09:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:09:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:09:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:09:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:09:41 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:09:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:09:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:09:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:09:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:09:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:09:41 [DEBUG   ] Finished updating board flags in 2.219 seconds
2018-11-20 12:09:45 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:10:05 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:10:15 [DEBUG   ] Got command RPT from MCS: ref #568538
2018-11-20 12:10:15 [DEBUG   ] summary = ERROR
2018-11-20 12:10:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568538  13 58442 43815005 A  ERRORERROR'
2018-11-20 12:10:15 [DEBUG   ] Got command RPT from MCS: ref #568539
2018-11-20 12:10:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:10:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568539  98 58442 43815197 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:10:15 [DEBUG   ] Got command RPT from MCS: ref #568540
2018-11-20 12:10:15 [DEBUG   ] summary = ERROR
2018-11-20 12:10:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568540  13 58442 43815401 A  ERRORERROR'
2018-11-20 12:10:15 [DEBUG   ] Got command RPT from MCS: ref #568542
2018-11-20 12:10:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:10:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568542  98 58442 43815605 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:10:15 [DEBUG   ] Got command RPT from MCS: ref #568543
2018-11-20 12:10:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:10:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568543  12 58442 43815801 A  ERRORB¢ff'
2018-11-20 12:10:25 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:10:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:10:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:10:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:10:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:10:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:10:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:10:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:10:41 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 12:10:41 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:10:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:10:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:10:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:10:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:10:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:10:41 [DEBUG   ] Finished updating board flags in 2.154 seconds
2018-11-20 12:10:45 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:10:46 [DEBUG   ] Got command RPT from MCS: ref #568552
2018-11-20 12:10:46 [DEBUG   ] summary = ERROR
2018-11-20 12:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568552  13 58442 43846701 A  ERRORERROR'
2018-11-20 12:10:46 [DEBUG   ] Got command RPT from MCS: ref #568553
2018-11-20 12:10:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568553  98 58442 43846702 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:11:05 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:11:25 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:11:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:11:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:11:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:11:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:11:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:11:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 12:11:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:11:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:11:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:11:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:11:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:11:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:11:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:11:41 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:11:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:11:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:11:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:11:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:11:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:11:41 [DEBUG   ] Finished updating board flags in 1.994 seconds
2018-11-20 12:11:45 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:12:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:12:25 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:12:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:12:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:12:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:12:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:12:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:12:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 12:12:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 12:12:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:12:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:12:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:12:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:12:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:12:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:12:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:12:41 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:12:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:12:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:12:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:12:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:12:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:12:41 [DEBUG   ] Finished updating board flags in 1.775 seconds
2018-11-20 12:12:45 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:13:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:13:25 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:13:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:13:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372340508000000
2018-11-20 12:13:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:13:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:13:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:13:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:13:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372340508000000
2018-11-20 12:13:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:13:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:13:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:13:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:13:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:13:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:13:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 12:13:41 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:13:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:13:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:13:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:13:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:13:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:13:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:13:41 [DEBUG   ] Finished updating board flags in 1.623 seconds
2018-11-20 12:13:45 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:14:05 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:14:25 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 12:14:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:14:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372352268000000
2018-11-20 12:14:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:14:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:14:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:14:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:14:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372352268000000
2018-11-20 12:14:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:14:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:14:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:14:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:14:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:14:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:14:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 12:14:41 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:14:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:14:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:14:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:14:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:14:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:14:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:14:41 [DEBUG   ] Finished updating board flags in 1.628 seconds
2018-11-20 12:14:45 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:15:05 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:15:15 [DEBUG   ] Got command RPT from MCS: ref #568573
2018-11-20 12:15:15 [DEBUG   ] summary = ERROR
2018-11-20 12:15:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568573  13 58442 44115081 A  ERRORERROR'
2018-11-20 12:15:15 [DEBUG   ] Got command RPT from MCS: ref #568574
2018-11-20 12:15:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:15:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568574  71 58442 44115285 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:15:15 [DEBUG   ] Got command RPT from MCS: ref #568575
2018-11-20 12:15:15 [DEBUG   ] summary = ERROR
2018-11-20 12:15:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568575  13 58442 44115477 A  ERRORERROR'
2018-11-20 12:15:15 [DEBUG   ] Got command RPT from MCS: ref #568576
2018-11-20 12:15:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:15:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568576  71 58442 44115681 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:15:15 [DEBUG   ] Got command RPT from MCS: ref #568577
2018-11-20 12:15:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:15:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568577  12 58442 44115885 A  ERRORB¡ff'
2018-11-20 12:15:25 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:15:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:15:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372364028000000
2018-11-20 12:15:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:15:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:15:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:15:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:15:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372364028000000
2018-11-20 12:15:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 12:15:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 12:15:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:15:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:15:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:15:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:15:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:15:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:15:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:15:41 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:15:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:15:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:15:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:15:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:15:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:15:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:15:41 [DEBUG   ] Finished updating board flags in 1.639 seconds
2018-11-20 12:15:45 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 12:15:46 [DEBUG   ] Got command RPT from MCS: ref #568578
2018-11-20 12:15:46 [DEBUG   ] summary = ERROR
2018-11-20 12:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568578  13 58442 44146701 A  ERRORERROR'
2018-11-20 12:15:46 [DEBUG   ] Got command RPT from MCS: ref #568579
2018-11-20 12:15:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568579  71 58442 44146702 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:16:05 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:16:25 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:16:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:16:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372375788000000
2018-11-20 12:16:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:16:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:16:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:16:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:16:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372375788000000
2018-11-20 12:16:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 12:16:40 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:16:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:16:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:16:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:16:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:16:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:16:41 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:16:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:16:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:16:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:16:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:16:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:16:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:16:41 [DEBUG   ] Finished updating board flags in 1.551 seconds
2018-11-20 12:16:45 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:17:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:17:25 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:17:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:17:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372387548000000
2018-11-20 12:17:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:17:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:17:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:17:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:17:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372387548000000
2018-11-20 12:17:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:17:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:17:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372387744000000
2018-11-20 12:17:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:17:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:17:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:17:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:17:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372387744000000
2018-11-20 12:17:41 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:17:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:17:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:17:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:17:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:17:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:17:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:17:41 [DEBUG   ] Finished updating board flags in 1.533 seconds
2018-11-20 12:17:45 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:18:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:18:25 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 12:18:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:18:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372399308000000
2018-11-20 12:18:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:18:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:18:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:18:40 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:18:40 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372399308000000
2018-11-20 12:18:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 12:18:40 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 12:18:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:18:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:18:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372399504000000
2018-11-20 12:18:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:18:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:18:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:18:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:18:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372399504000000
2018-11-20 12:18:41 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:18:41 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 12:18:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:18:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:18:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:18:41 [WARNING ] (13, 1, 'bot')
2018-11-20 12:18:41 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:18:41 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:18:41 [DEBUG   ] Finished updating board flags in 1.550 seconds
2018-11-20 12:18:45 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:19:05 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:19:25 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:19:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:19:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:19:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:19:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:19:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:19:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 12:19:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:19:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:19:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:19:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:19:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:19:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:19:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa21, chip 1
2018-11-20 12:19:42 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:19:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:19:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:19:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:19:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:19:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:19:42 [DEBUG   ] Finished updating board flags in 2.156 seconds
2018-11-20 12:19:45 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:20:05 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:20:15 [DEBUG   ] Got command RPT from MCS: ref #568605
2018-11-20 12:20:15 [DEBUG   ] summary = ERROR
2018-11-20 12:20:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568605  13 58442 44415161 A  ERRORERROR'
2018-11-20 12:20:15 [DEBUG   ] Got command RPT from MCS: ref #568606
2018-11-20 12:20:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:20:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568606  98 58442 44415365 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:20:15 [DEBUG   ] Got command RPT from MCS: ref #568607
2018-11-20 12:20:15 [DEBUG   ] summary = ERROR
2018-11-20 12:20:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568607  13 58442 44415557 A  ERRORERROR'
2018-11-20 12:20:15 [DEBUG   ] Got command RPT from MCS: ref #568609
2018-11-20 12:20:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:20:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568609  98 58442 44415761 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:20:15 [DEBUG   ] Got command RPT from MCS: ref #568610
2018-11-20 12:20:15 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:20:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568610  12 58442 44415965 A  ERRORB¡ff'
2018-11-20 12:20:25 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:20:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:20:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:20:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:20:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:20:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:20:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 12:20:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:20:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:20:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:20:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:20:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:20:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:20:42 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:20:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 12:20:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:20:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:20:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:20:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:20:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:20:42 [DEBUG   ] Finished updating board flags in 2.037 seconds
2018-11-20 12:20:45 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:20:46 [DEBUG   ] Got command RPT from MCS: ref #568615
2018-11-20 12:20:46 [DEBUG   ] summary = ERROR
2018-11-20 12:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568615  13 58442 44446705 A  ERRORERROR'
2018-11-20 12:20:46 [DEBUG   ] Got command RPT from MCS: ref #568616
2018-11-20 12:20:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568616  98 58442 44446706 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:21:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:21:25 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:21:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:21:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:21:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:21:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:21:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:21:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 12:21:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 3
2018-11-20 12:21:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:21:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:21:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:21:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:21:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:21:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:21:42 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:21:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:21:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:21:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:21:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:21:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:21:42 [DEBUG   ] Finished updating board flags in 1.936 seconds
2018-11-20 12:21:45 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:22:05 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:22:25 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:22:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:22:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:22:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:22:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:22:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:22:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 12:22:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 12:22:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:22:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:22:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:22:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:22:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:22:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:22:42 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:22:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:22:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:22:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:22:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:22:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:22:42 [DEBUG   ] Finished updating board flags in 1.780 seconds
2018-11-20 12:22:45 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:23:05 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:23:25 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:23:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:23:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372458304000000
2018-11-20 12:23:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:23:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:23:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:23:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:23:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372458304000000
2018-11-20 12:23:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 12:23:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 1
2018-11-20 12:23:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:23:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:23:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:23:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:23:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:23:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:23:42 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:23:42 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 12:23:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:23:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:23:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:23:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:23:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:23:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:23:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:23:42 [DEBUG   ] Finished updating board flags in 1.587 seconds
2018-11-20 12:23:45 [DEBUG   ] Finished updating ant stats in 0.595 seconds
2018-11-20 12:24:06 [DEBUG   ] Finished updating ant stats in 1.398 seconds
2018-11-20 12:24:26 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 12:24:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:24:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372470064000000
2018-11-20 12:24:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:24:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:24:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:24:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:24:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372470064000000
2018-11-20 12:24:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:24:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 12:24:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:24:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:24:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:24:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:24:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:24:42 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 12:24:42 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:24:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:24:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:24:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:24:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:24:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:24:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:24:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:24:42 [DEBUG   ] Finished updating board flags in 1.428 seconds
2018-11-20 12:24:46 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 12:25:06 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 12:25:15 [DEBUG   ] Got command RPT from MCS: ref #568636
2018-11-20 12:25:15 [DEBUG   ] summary = ERROR
2018-11-20 12:25:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568636  13 58442 44715237 A  ERRORERROR'
2018-11-20 12:25:15 [DEBUG   ] Got command RPT from MCS: ref #568637
2018-11-20 12:25:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:25:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568637  71 58442 44715441 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:25:15 [DEBUG   ] Got command RPT from MCS: ref #568638
2018-11-20 12:25:15 [DEBUG   ] summary = ERROR
2018-11-20 12:25:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568638  13 58442 44715645 A  ERRORERROR'
2018-11-20 12:25:15 [DEBUG   ] Got command RPT from MCS: ref #568639
2018-11-20 12:25:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:25:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568639  71 58442 44715837 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:25:16 [DEBUG   ] Got command RPT from MCS: ref #568640
2018-11-20 12:25:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:25:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568640  12 58442 44716041 A  ERRORB£ff'
2018-11-20 12:25:26 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:25:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:25:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372481824000000
2018-11-20 12:25:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:25:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:25:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:25:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:25:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372481824000000
2018-11-20 12:25:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 12:25:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 12:25:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:25:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:25:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:25:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:25:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:25:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:25:42 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:25:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:25:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:25:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:25:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:25:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:25:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:25:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:25:42 [DEBUG   ] Finished updating board flags in 1.428 seconds
2018-11-20 12:25:46 [DEBUG   ] Finished updating ant stats in 1.199 seconds
2018-11-20 12:25:46 [DEBUG   ] Got command RPT from MCS: ref #568645
2018-11-20 12:25:46 [DEBUG   ] summary = ERROR
2018-11-20 12:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568645  13 58442 44746705 A  ERRORERROR'
2018-11-20 12:25:46 [DEBUG   ] Got command RPT from MCS: ref #568646
2018-11-20 12:25:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:25:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568646  71 58442 44746706 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:26:06 [DEBUG   ] Finished updating ant stats in 1.004 seconds
2018-11-20 12:26:26 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 12:26:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:26:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372493584000000
2018-11-20 12:26:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:26:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:26:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:26:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:26:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372493584000000
2018-11-20 12:26:41 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:26:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:26:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:26:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:26:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:26:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:26:42 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:26:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:26:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:26:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:26:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:26:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:26:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:26:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:26:42 [DEBUG   ] Finished updating board flags in 1.435 seconds
2018-11-20 12:26:46 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 12:27:06 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 12:27:26 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 12:27:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:27:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372505344000000
2018-11-20 12:27:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:27:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:27:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:27:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:27:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372505344000000
2018-11-20 12:27:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 12:27:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 12:27:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:27:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:27:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372505540000000
2018-11-20 12:27:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:27:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:27:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:27:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:27:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372505540000000
2018-11-20 12:27:42 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:27:42 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 12:27:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:27:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:27:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:27:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:27:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:27:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:27:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:27:42 [DEBUG   ] Finished updating board flags in 1.469 seconds
2018-11-20 12:27:46 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 12:28:06 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 12:28:26 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 12:28:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:28:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372517104000000
2018-11-20 12:28:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:28:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:28:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:28:41 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:28:41 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372517104000000
2018-11-20 12:28:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 12:28:41 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 5
2018-11-20 12:28:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:28:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:28:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372517300000000
2018-11-20 12:28:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:28:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:28:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:28:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:28:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372517300000000
2018-11-20 12:28:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:28:42 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:28:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:28:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:28:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:28:42 [WARNING ] (13, 1, 'bot')
2018-11-20 12:28:42 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:28:42 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:28:42 [DEBUG   ] Finished updating board flags in 1.492 seconds
2018-11-20 12:28:46 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 12:29:06 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 12:29:26 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 12:29:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:29:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:29:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:29:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:29:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:29:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:29:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:29:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 12:29:43 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:29:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:29:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:29:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:29:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:29:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:29:43 [DEBUG   ] Finished updating board flags in 2.251 seconds
2018-11-20 12:29:46 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 12:30:06 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 12:30:15 [DEBUG   ] Got command RPT from MCS: ref #568668
2018-11-20 12:30:15 [DEBUG   ] summary = ERROR
2018-11-20 12:30:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568668  13 58442 45015317 A  ERRORERROR'
2018-11-20 12:30:15 [DEBUG   ] Got command RPT from MCS: ref #568669
2018-11-20 12:30:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:30:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568669  98 58442 45015521 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:30:15 [DEBUG   ] Got command RPT from MCS: ref #568670
2018-11-20 12:30:15 [DEBUG   ] summary = ERROR
2018-11-20 12:30:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568670  13 58442 45015725 A  ERRORERROR'
2018-11-20 12:30:15 [DEBUG   ] Got command RPT from MCS: ref #568672
2018-11-20 12:30:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:30:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568672  98 58442 45015917 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:30:16 [DEBUG   ] Got command RPT from MCS: ref #568673
2018-11-20 12:30:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:30:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568673  12 58442 45016117 A  ERRORB£ff'
2018-11-20 12:30:26 [DEBUG   ] Finished updating ant stats in 0.998 seconds
2018-11-20 12:30:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:30:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:30:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:30:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:30:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:30:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 12:30:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:30:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:30:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:30:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:30:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:30:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:30:43 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:30:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:30:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:30:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:30:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:30:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:30:43 [DEBUG   ] Finished updating board flags in 2.215 seconds
2018-11-20 12:30:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:30:46 [DEBUG   ] Got command RPT from MCS: ref #568678
2018-11-20 12:30:46 [DEBUG   ] summary = ERROR
2018-11-20 12:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568678  13 58442 45046701 A  ERRORERROR'
2018-11-20 12:30:46 [DEBUG   ] Got command RPT from MCS: ref #568679
2018-11-20 12:30:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:30:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568679  98 58442 45046702 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:31:06 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:31:26 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:31:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:31:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:31:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:31:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:31:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:31:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 12:31:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:31:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:31:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:31:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:31:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:31:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:31:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:31:43 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:31:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:31:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:31:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:31:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:31:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:31:43 [DEBUG   ] Finished updating board flags in 2.105 seconds
2018-11-20 12:31:46 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:32:06 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:32:26 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:32:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:32:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:32:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:32:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:32:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:32:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:32:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:32:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:32:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:32:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:32:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:32:43 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:32:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:32:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:32:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:32:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:32:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:32:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:32:43 [DEBUG   ] Finished updating board flags in 2.043 seconds
2018-11-20 12:32:46 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:33:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:33:26 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:33:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:33:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:33:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:33:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:33:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:33:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:33:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:33:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:33:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:33:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:33:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:33:43 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:33:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:33:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:33:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:33:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:33:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:33:43 [DEBUG   ] Finished updating board flags in 1.964 seconds
2018-11-20 12:33:46 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:34:06 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:34:26 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:34:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:34:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:34:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:34:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:34:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:34:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 12:34:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:34:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 12:34:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:34:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:34:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:34:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:34:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:34:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:34:43 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:34:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:34:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:34:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:34:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:34:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:34:43 [DEBUG   ] Finished updating board flags in 1.811 seconds
2018-11-20 12:34:46 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:35:06 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:35:15 [DEBUG   ] Got command RPT from MCS: ref #568703
2018-11-20 12:35:15 [DEBUG   ] summary = ERROR
2018-11-20 12:35:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568703  13 58442 45315405 A  ERRORERROR'
2018-11-20 12:35:15 [DEBUG   ] Got command RPT from MCS: ref #568704
2018-11-20 12:35:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:35:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568704  98 58442 45315597 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:35:15 [DEBUG   ] Got command RPT from MCS: ref #568705
2018-11-20 12:35:15 [DEBUG   ] summary = ERROR
2018-11-20 12:35:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568705  13 58442 45315801 A  ERRORERROR'
2018-11-20 12:35:16 [DEBUG   ] Got command RPT from MCS: ref #568706
2018-11-20 12:35:16 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:35:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568706  98 58442 45316005 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:35:16 [DEBUG   ] Got command RPT from MCS: ref #568707
2018-11-20 12:35:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:35:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568707  12 58442 45316197 A  ERRORB¢ff'
2018-11-20 12:35:26 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 12:35:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:35:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:35:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:35:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:35:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:35:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:35:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:35:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:35:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:35:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:35:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:35:43 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:35:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:35:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:35:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:35:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:35:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:35:43 [DEBUG   ] Finished updating board flags in 1.825 seconds
2018-11-20 12:35:46 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:35:46 [DEBUG   ] Got command RPT from MCS: ref #568708
2018-11-20 12:35:46 [DEBUG   ] summary = ERROR
2018-11-20 12:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568708  13 58442 45346705 A  ERRORERROR'
2018-11-20 12:35:46 [DEBUG   ] Got command RPT from MCS: ref #568709
2018-11-20 12:35:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:35:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568709  98 58442 45346706 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:36:06 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:36:26 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:36:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:36:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:36:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:36:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:36:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:36:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:36:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:36:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:36:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:36:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:36:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:36:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:36:43 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:36:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:36:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:36:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:36:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:36:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:36:43 [DEBUG   ] Finished updating board flags in 1.854 seconds
2018-11-20 12:36:46 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:37:06 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:37:26 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:37:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:37:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:37:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:37:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:37:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:37:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:37:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:37:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:37:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:37:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:37:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:37:43 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:37:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 12:37:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:37:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:37:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:37:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:37:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:37:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:37:43 [DEBUG   ] Finished updating board flags in 1.736 seconds
2018-11-20 12:37:46 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:38:06 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:38:26 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:38:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:38:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372634900000000
2018-11-20 12:38:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:38:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:38:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:38:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:38:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372634900000000
2018-11-20 12:38:42 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 12:38:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 12:38:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa11, chip 5
2018-11-20 12:38:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:38:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:38:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:38:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:38:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:38:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:38:43 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:38:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:38:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:38:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:38:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:38:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:38:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:38:43 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:38:43 [DEBUG   ] Finished updating board flags in 1.644 seconds
2018-11-20 12:38:46 [DEBUG   ] Finished updating ant stats in 0.797 seconds
2018-11-20 12:39:06 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:39:26 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:39:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:39:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372646660000000
2018-11-20 12:39:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:39:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:39:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:39:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:39:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372646660000000
2018-11-20 12:39:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:39:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:39:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:39:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:39:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:39:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:39:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 12:39:43 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:39:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:39:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:39:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:39:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:39:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:39:43 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:39:43 [DEBUG   ] Finished updating board flags in 1.557 seconds
2018-11-20 12:39:46 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:40:06 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:40:15 [DEBUG   ] Got command RPT from MCS: ref #568731
2018-11-20 12:40:15 [DEBUG   ] summary = ERROR
2018-11-20 12:40:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568731  13 58442 45615481 A  ERRORERROR'
2018-11-20 12:40:15 [DEBUG   ] Got command RPT from MCS: ref #568732
2018-11-20 12:40:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:40:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568732  71 58442 45615685 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:40:15 [DEBUG   ] Got command RPT from MCS: ref #568733
2018-11-20 12:40:15 [DEBUG   ] summary = ERROR
2018-11-20 12:40:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568733  13 58442 45615877 A  ERRORERROR'
2018-11-20 12:40:16 [DEBUG   ] Got command RPT from MCS: ref #568735
2018-11-20 12:40:16 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:40:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568735  71 58442 45616081 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:40:16 [DEBUG   ] Got command RPT from MCS: ref #568737
2018-11-20 12:40:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:40:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568737  12 58442 45616285 A  ERRORB¡ff'
2018-11-20 12:40:26 [DEBUG   ] Finished updating ant stats in 0.599 seconds
2018-11-20 12:40:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:40:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372658420000000
2018-11-20 12:40:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:40:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:40:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:40:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:40:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372658420000000
2018-11-20 12:40:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:40:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:40:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:40:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:40:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:40:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:40:43 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:40:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:40:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:40:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:40:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:40:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:40:43 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:40:43 [DEBUG   ] Finished updating board flags in 1.452 seconds
2018-11-20 12:40:46 [DEBUG   ] Got command RPT from MCS: ref #568745
2018-11-20 12:40:46 [DEBUG   ] summary = ERROR
2018-11-20 12:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568745  13 58442 45646709 A  ERRORERROR'
2018-11-20 12:40:46 [DEBUG   ] Got command RPT from MCS: ref #568746
2018-11-20 12:40:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:40:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568746  71 58442 45646710 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:40:47 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-20 12:41:07 [DEBUG   ] Finished updating ant stats in 1.205 seconds
2018-11-20 12:41:27 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 12:41:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:41:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372670180000000
2018-11-20 12:41:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:41:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:41:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:41:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:41:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372670180000000
2018-11-20 12:41:42 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 12:41:42 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:41:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:41:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372670376000000
2018-11-20 12:41:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:41:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:41:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:41:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:41:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372670376000000
2018-11-20 12:41:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:41:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 12:41:43 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:41:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 12:41:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:41:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:41:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:41:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:41:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:41:43 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:41:43 [DEBUG   ] Finished updating board flags in 1.437 seconds
2018-11-20 12:41:47 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:42:07 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 12:42:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:42:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:42:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372681940000000
2018-11-20 12:42:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:42:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:42:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:42:42 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:42:42 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372681940000000
2018-11-20 12:42:42 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 12:42:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:42:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:42:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372682136000000
2018-11-20 12:42:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:42:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:42:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:42:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:42:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372682136000000
2018-11-20 12:42:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 12:42:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 12:42:43 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:42:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:42:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:42:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:42:43 [WARNING ] (13, 1, 'bot')
2018-11-20 12:42:43 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:42:43 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:42:43 [DEBUG   ] Finished updating board flags in 1.518 seconds
2018-11-20 12:42:47 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 12:43:07 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:43:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:43:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:43:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:43:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:43:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:43:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:43:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:43:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:43:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:43:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:43:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:43:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:43:44 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:43:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 12:43:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:43:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:43:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:43:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:43:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:43:44 [DEBUG   ] Finished updating board flags in 2.250 seconds
2018-11-20 12:43:47 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 12:44:07 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 12:44:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:44:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:44:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:44:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:44:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:44:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:44:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 12:44:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:44:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:44:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:44:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:44:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:44:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:44:44 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 12:44:44 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:44:44 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 12:44:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:44:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:44:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:44:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:44:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:44:44 [DEBUG   ] Finished updating board flags in 2.131 seconds
2018-11-20 12:44:47 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:45:07 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 12:45:15 [DEBUG   ] Got command RPT from MCS: ref #568766
2018-11-20 12:45:15 [DEBUG   ] summary = ERROR
2018-11-20 12:45:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568766  13 58442 45915561 A  ERRORERROR'
2018-11-20 12:45:15 [DEBUG   ] Got command RPT from MCS: ref #568767
2018-11-20 12:45:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:45:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568767  98 58442 45915765 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:45:15 [DEBUG   ] Got command RPT from MCS: ref #568768
2018-11-20 12:45:15 [DEBUG   ] summary = ERROR
2018-11-20 12:45:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568768  13 58442 45915969 A  ERRORERROR'
2018-11-20 12:45:16 [DEBUG   ] Got command RPT from MCS: ref #568769
2018-11-20 12:45:16 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:45:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568769  98 58442 45916161 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:45:16 [DEBUG   ] Got command RPT from MCS: ref #568770
2018-11-20 12:45:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:45:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568770  12 58442 45916365 A  ERRORB¡ff'
2018-11-20 12:45:27 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 12:45:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:45:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:45:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:45:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:45:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:45:43 [WARNING ] MonitorFlags: VCCINT 0.979 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 12:45:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:45:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:45:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:45:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:45:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:45:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:45:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 12:45:44 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:45:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:45:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:45:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:45:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:45:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:45:44 [DEBUG   ] Finished updating board flags in 2.062 seconds
2018-11-20 12:45:46 [DEBUG   ] Got command RPT from MCS: ref #568771
2018-11-20 12:45:46 [DEBUG   ] summary = ERROR
2018-11-20 12:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568771  13 58442 45946709 A  ERRORERROR'
2018-11-20 12:45:46 [DEBUG   ] Got command RPT from MCS: ref #568772
2018-11-20 12:45:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:45:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568772  98 58442 45946710 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:45:47 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:46:07 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 12:46:27 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:46:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:46:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:46:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:46:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:46:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:46:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 12:46:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 12:46:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:46:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:46:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:46:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:46:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:46:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:46:44 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:46:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:46:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:46:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:46:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:46:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:46:44 [DEBUG   ] Finished updating board flags in 1.966 seconds
2018-11-20 12:46:47 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 12:47:07 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 12:47:27 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 12:47:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:47:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:47:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:47:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:47:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:47:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:47:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:47:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:47:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:47:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:47:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:47:44 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:47:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:47:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:47:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:47:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:47:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:47:44 [DEBUG   ] Finished updating board flags in 1.817 seconds
2018-11-20 12:47:47 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 12:48:07 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 12:48:27 [DEBUG   ] Finished updating ant stats in 1.198 seconds
2018-11-20 12:48:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:48:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:48:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:48:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:48:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:48:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:48:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:48:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:48:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:48:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:48:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:48:44 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:48:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:48:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:48:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:48:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:48:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:48:44 [DEBUG   ] Finished updating board flags in 1.780 seconds
2018-11-20 12:48:47 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 12:49:07 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 12:49:27 [DEBUG   ] Finished updating ant stats in 1.003 seconds
2018-11-20 12:49:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:49:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372764456000000
2018-11-20 12:49:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:49:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:49:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:49:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:49:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372764456000000
2018-11-20 12:49:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:49:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:49:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:49:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:49:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:49:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:49:44 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:49:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:49:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:49:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:49:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:49:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:49:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:49:44 [DEBUG   ] Finished updating board flags in 1.633 seconds
2018-11-20 12:49:47 [DEBUG   ] Finished updating ant stats in 1.002 seconds
2018-11-20 12:50:07 [DEBUG   ] Finished updating ant stats in 1.001 seconds
2018-11-20 12:50:15 [DEBUG   ] Got command RPT from MCS: ref #568798
2018-11-20 12:50:15 [DEBUG   ] summary = ERROR
2018-11-20 12:50:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568798  13 58442 46215645 A  ERRORERROR'
2018-11-20 12:50:15 [DEBUG   ] Got command RPT from MCS: ref #568799
2018-11-20 12:50:15 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:50:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568799  71 58442 46215849 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:50:16 [DEBUG   ] Got command RPT from MCS: ref #568800
2018-11-20 12:50:16 [DEBUG   ] summary = ERROR
2018-11-20 12:50:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568800  13 58442 46216053 A  ERRORERROR'
2018-11-20 12:50:16 [DEBUG   ] Got command RPT from MCS: ref #568802
2018-11-20 12:50:16 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:50:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568802  71 58442 46216245 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:50:16 [DEBUG   ] Got command RPT from MCS: ref #568803
2018-11-20 12:50:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:50:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568803  12 58442 46216449 A  ERRORB¡ff'
2018-11-20 12:50:27 [DEBUG   ] Finished updating ant stats in 0.997 seconds
2018-11-20 12:50:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:50:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372776216000000
2018-11-20 12:50:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:50:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:50:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:50:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:50:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372776216000000
2018-11-20 12:50:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 12:50:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:50:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 12:50:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:50:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:50:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:50:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:50:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:50:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 12:50:44 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:50:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:50:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:50:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:50:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:50:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:50:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:50:44 [DEBUG   ] Finished updating board flags in 1.651 seconds
2018-11-20 12:50:46 [DEBUG   ] Got command RPT from MCS: ref #568808
2018-11-20 12:50:46 [DEBUG   ] summary = ERROR
2018-11-20 12:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568808  13 58442 46246705 A  ERRORERROR'
2018-11-20 12:50:46 [DEBUG   ] Got command RPT from MCS: ref #568809
2018-11-20 12:50:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 12:50:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568809  71 58442 46246706 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 12:50:47 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 12:51:07 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:51:27 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:51:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:51:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372787976000000
2018-11-20 12:51:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:51:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:51:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:51:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:51:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372787976000000
2018-11-20 12:51:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 12:51:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:51:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:51:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:51:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:51:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:51:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:51:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 12:51:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 12:51:44 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:51:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:51:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:51:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:51:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:51:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:51:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 12:51:44 [DEBUG   ] Finished updating board flags in 1.502 seconds
2018-11-20 12:51:47 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:52:07 [DEBUG   ] Finished updating ant stats in 0.803 seconds
2018-11-20 12:52:27 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:52:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:52:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372799736000000
2018-11-20 12:52:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:52:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:52:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:52:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:52:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372799736000000
2018-11-20 12:52:43 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 12:52:43 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:52:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:52:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372799932000000
2018-11-20 12:52:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:52:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:52:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:52:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:52:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372799932000000
2018-11-20 12:52:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:52:44 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:52:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:52:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:52:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:52:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:52:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:52:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:52:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:52:44 [DEBUG   ] Finished updating board flags in 1.508 seconds
2018-11-20 12:52:47 [DEBUG   ] Finished updating ant stats in 0.801 seconds
2018-11-20 12:53:07 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:53:27 [DEBUG   ] Finished updating ant stats in 0.804 seconds
2018-11-20 12:53:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:53:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372811496000000
2018-11-20 12:53:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:53:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:53:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:53:43 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:53:43 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372811496000000
2018-11-20 12:53:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:53:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:53:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372811692000000
2018-11-20 12:53:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:53:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:53:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:53:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:53:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372811692000000
2018-11-20 12:53:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:53:44 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:53:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:53:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:53:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:53:44 [WARNING ] (13, 1, 'bot')
2018-11-20 12:53:44 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:53:44 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 12:53:44 [DEBUG   ] Finished updating board flags in 1.565 seconds
2018-11-20 12:53:47 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:54:07 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:54:27 [DEBUG   ] Finished updating ant stats in 0.802 seconds
2018-11-20 12:54:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:54:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:54:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:54:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:54:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:54:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:54:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:54:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:54:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:54:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:54:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:54:45 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 12:54:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 12:54:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:54:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:54:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:54:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:54:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:54:45 [DEBUG   ] Finished updating board flags in 2.219 seconds
2018-11-20 12:54:47 [DEBUG   ] Finished updating ant stats in 0.800 seconds
2018-11-20 12:55:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:55:15 [DEBUG   ] Got command RPT from MCS: ref #568829
2018-11-20 12:55:15 [DEBUG   ] summary = ERROR
2018-11-20 12:55:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568829  13 58442 46515717 A  ERRORERROR'
2018-11-20 12:55:15 [DEBUG   ] Got command RPT from MCS: ref #568830
2018-11-20 12:55:15 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:55:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568830  98 58442 46515921 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:55:16 [DEBUG   ] Got command RPT from MCS: ref #568831
2018-11-20 12:55:16 [DEBUG   ] summary = ERROR
2018-11-20 12:55:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568831  13 58442 46516125 A  ERRORERROR'
2018-11-20 12:55:16 [DEBUG   ] Got command RPT from MCS: ref #568832
2018-11-20 12:55:16 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:55:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568832  98 58442 46516329 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:55:16 [DEBUG   ] Got command RPT from MCS: ref #568833
2018-11-20 12:55:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 12:55:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568833  12 58442 46516521 A  ERRORB¢ff'
2018-11-20 12:55:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:55:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:55:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:55:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:55:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:55:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:55:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa8, chip 2
2018-11-20 12:55:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:55:44 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 12:55:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:55:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:55:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:55:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:55:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:55:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 12:55:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 12:55:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:55:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:55:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:55:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:55:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:55:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:55:45 [DEBUG   ] Finished updating board flags in 2.211 seconds
2018-11-20 12:55:46 [DEBUG   ] Got command RPT from MCS: ref #568838
2018-11-20 12:55:46 [DEBUG   ] summary = ERROR
2018-11-20 12:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568838  13 58442 46546713 A  ERRORERROR'
2018-11-20 12:55:46 [DEBUG   ] Got command RPT from MCS: ref #568839
2018-11-20 12:55:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 12:55:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568839  98 58442 46546715 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 12:55:47 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:56:07 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:56:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:56:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:56:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:56:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:56:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:56:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:56:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 12:56:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:56:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:56:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:56:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:56:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:56:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:56:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:56:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:56:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:56:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:56:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:56:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:56:45 [DEBUG   ] Finished updating board flags in 2.154 seconds
2018-11-20 12:56:47 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:57:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:57:27 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 12:57:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:57:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:57:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:57:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:57:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:57:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:57:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:57:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:57:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:57:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:57:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:57:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:57:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:57:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:57:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:57:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:57:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:57:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:57:45 [DEBUG   ] Finished updating board flags in 2.019 seconds
2018-11-20 12:57:47 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:58:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:58:27 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:58:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:58:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:58:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:58:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:58:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:58:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:58:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:58:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:58:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:58:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:58:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:58:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:58:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 12:58:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 12:58:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:58:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:58:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:58:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:58:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:58:45 [DEBUG   ] Finished updating board flags in 1.938 seconds
2018-11-20 12:58:47 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 12:59:07 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 12:59:27 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 12:59:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 12:59:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 12:59:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 12:59:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 12:59:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 12:59:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 12:59:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 12:59:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 12:59:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 12:59:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 12:59:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 12:59:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 12:59:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:59:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:59:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:59:45 [WARNING ] (13, 1, 'bot')
2018-11-20 12:59:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 12:59:45 [DEBUG   ] Finished updating board flags in 1.819 seconds
2018-11-20 12:59:47 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:00:07 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 13:00:15 [DEBUG   ] Got command RPT from MCS: ref #568861
2018-11-20 13:00:15 [DEBUG   ] summary = ERROR
2018-11-20 13:00:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568861  13 58442 46815801 A  ERRORERROR'
2018-11-20 13:00:16 [DEBUG   ] Got command RPT from MCS: ref #568862
2018-11-20 13:00:16 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 13:00:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568862  98 58442 46816005 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 13:00:16 [DEBUG   ] Got command RPT from MCS: ref #568863
2018-11-20 13:00:16 [DEBUG   ] summary = ERROR
2018-11-20 13:00:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568863  13 58442 46816209 A  ERRORERROR'
2018-11-20 13:00:16 [DEBUG   ] Got command RPT from MCS: ref #568864
2018-11-20 13:00:16 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 13:00:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568864  98 58442 46816401 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 13:00:16 [DEBUG   ] Got command RPT from MCS: ref #568866
2018-11-20 13:00:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 13:00:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568866  12 58442 46816605 A  ERRORB£ff'
2018-11-20 13:00:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:00:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:00:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372894012000000
2018-11-20 13:00:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:00:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:00:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:00:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:00:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372894012000000
2018-11-20 13:00:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:00:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 13:00:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa14, chip 4
2018-11-20 13:00:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:00:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:00:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:00:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:00:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:00:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:00:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:00:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:00:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:00:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:00:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:00:45 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:00:45 [DEBUG   ] Finished updating board flags in 1.870 seconds
2018-11-20 13:00:46 [DEBUG   ] Got command RPT from MCS: ref #568871
2018-11-20 13:00:46 [DEBUG   ] summary = ERROR
2018-11-20 13:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568871  13 58442 46846705 A  ERRORERROR'
2018-11-20 13:00:46 [DEBUG   ] Got command RPT from MCS: ref #568872
2018-11-20 13:00:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 13:00:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568872  71 58442 46846706 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 13:00:47 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:01:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:01:27 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 13:01:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:01:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:01:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:01:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:01:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:01:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 13:01:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:01:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:01:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:01:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:01:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:01:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:01:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:01:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 13:01:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:01:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:01:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:01:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:01:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:01:45 [DEBUG   ] Finished updating board flags in 1.726 seconds
2018-11-20 13:01:47 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:02:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:02:27 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:02:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:02:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372917532000000
2018-11-20 13:02:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:02:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:02:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:02:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:02:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372917532000000
2018-11-20 13:02:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 13:02:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:02:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:02:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:02:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:02:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:02:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:02:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:02:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:02:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:02:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:02:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:02:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:02:45 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:02:45 [DEBUG   ] Finished updating board flags in 1.642 seconds
2018-11-20 13:02:47 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 13:03:07 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:03:27 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:03:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:03:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372929292000000
2018-11-20 13:03:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:03:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:03:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:03:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:03:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372929292000000
2018-11-20 13:03:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 13:03:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:03:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:03:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:03:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:03:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:03:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:03:45 [WARNING ] MonitorFlags: Temperature 81.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:03:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:03:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:03:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:03:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:03:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:03:45 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:03:45 [DEBUG   ] Finished updating board flags in 1.576 seconds
2018-11-20 13:03:47 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:04:07 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 13:04:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:04:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:04:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372941052000000
2018-11-20 13:04:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:04:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:04:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:04:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:04:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372941052000000
2018-11-20 13:04:44 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 13:04:44 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:04:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:04:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:04:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:04:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:04:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:04:45 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:04:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa33, chip 4
2018-11-20 13:04:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:04:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:04:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:04:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:04:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:04:45 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:04:45 [DEBUG   ] Finished updating board flags in 1.416 seconds
2018-11-20 13:04:47 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:05:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:05:15 [DEBUG   ] Got command RPT from MCS: ref #568896
2018-11-20 13:05:15 [DEBUG   ] summary = ERROR
2018-11-20 13:05:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568896  13 58442 47115885 A  ERRORERROR'
2018-11-20 13:05:16 [DEBUG   ] Got command RPT from MCS: ref #568897
2018-11-20 13:05:16 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 13:05:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568897  71 58442 47116089 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 13:05:16 [DEBUG   ] Got command RPT from MCS: ref #568898
2018-11-20 13:05:16 [DEBUG   ] summary = ERROR
2018-11-20 13:05:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568898  13 58442 47116281 A  ERRORERROR'
2018-11-20 13:05:16 [DEBUG   ] Got command RPT from MCS: ref #568899
2018-11-20 13:05:16 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3
2018-11-20 13:05:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568899  71 58442 47116485 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3'
2018-11-20 13:05:16 [DEBUG   ] Got command RPT from MCS: ref #568900
2018-11-20 13:05:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 13:05:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568900  12 58442 47116689 A  ERRORB¢ff'
2018-11-20 13:05:27 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 13:05:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:05:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372952812000000
2018-11-20 13:05:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:05:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:05:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:05:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:05:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372952812000000
2018-11-20 13:05:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:05:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:05:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372953008000000
2018-11-20 13:05:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:05:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:05:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:05:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:05:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372953008000000
2018-11-20 13:05:45 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:05:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa15, chip 2
2018-11-20 13:05:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:05:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:05:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:05:45 [WARNING ] (13, 1, 'bot')
2018-11-20 13:05:45 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:05:45 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 13:05:45 [DEBUG   ] Finished updating board flags in 1.554 seconds
2018-11-20 13:05:46 [DEBUG   ] Got command RPT from MCS: ref #568901
2018-11-20 13:05:46 [DEBUG   ] summary = ERROR
2018-11-20 13:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568901  13 58442 47146705 A  ERRORERROR'
2018-11-20 13:05:46 [DEBUG   ] Got command RPT from MCS: ref #568902
2018-11-20 13:05:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 13:05:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568902  79 58442 47146706 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 13:05:47 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:06:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:06:27 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:06:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:06:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302372964572000000
2018-11-20 13:06:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:06:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:06:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:06:44 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:06:44 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302372964572000000
2018-11-20 13:06:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:06:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 3
2018-11-20 13:06:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:06:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302372964768000000
2018-11-20 13:06:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:06:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:06:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:06:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:06:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302372964768000000
2018-11-20 13:06:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 13:06:45 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:06:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:06:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:06:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:06:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:06:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:06:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 13:06:46 [DEBUG   ] Finished updating board flags in 1.572 seconds
2018-11-20 13:06:47 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 13:07:07 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:07:27 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:07:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:07:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:07:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:07:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:07:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:07:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:07:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:07:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:07:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:07:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:07:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:07:46 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:07:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:07:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:07:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:07:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:07:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:07:46 [DEBUG   ] Finished updating board flags in 2.159 seconds
2018-11-20 13:07:47 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:08:07 [DEBUG   ] Finished updating ant stats in 0.603 seconds
2018-11-20 13:08:27 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:08:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:08:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:08:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:08:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:08:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:08:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 13:08:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:08:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:08:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:08:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:08:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:08:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:08:46 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:08:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:08:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:08:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:08:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:08:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:08:46 [DEBUG   ] Finished updating board flags in 1.914 seconds
2018-11-20 13:08:47 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:09:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:09:27 [DEBUG   ] Finished updating ant stats in 0.604 seconds
2018-11-20 13:09:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:09:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:09:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:09:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:09:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:09:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 13:09:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 13:09:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:09:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:09:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:09:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:09:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:09:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:09:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 13:09:46 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:09:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:09:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:09:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:09:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:09:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:09:46 [DEBUG   ] Finished updating board flags in 1.830 seconds
2018-11-20 13:09:47 [DEBUG   ] Finished updating ant stats in 0.602 seconds
2018-11-20 13:10:07 [DEBUG   ] Finished updating ant stats in 0.601 seconds
2018-11-20 13:10:15 [DEBUG   ] Got command RPT from MCS: ref #568924
2018-11-20 13:10:15 [DEBUG   ] summary = ERROR
2018-11-20 13:10:15 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568924  13 58442 47415957 A  ERRORERROR'
2018-11-20 13:10:16 [DEBUG   ] Got command RPT from MCS: ref #568925
2018-11-20 13:10:16 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 13:10:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568925  98 58442 47416161 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 13:10:16 [DEBUG   ] Got command RPT from MCS: ref #568926
2018-11-20 13:10:16 [DEBUG   ] summary = ERROR
2018-11-20 13:10:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568926  13 58442 47416365 A  ERRORERROR'
2018-11-20 13:10:16 [DEBUG   ] Got command RPT from MCS: ref #568927
2018-11-20 13:10:16 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 13:10:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568927  98 58442 47416557 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 13:10:16 [DEBUG   ] Got command RPT from MCS: ref #568929
2018-11-20 13:10:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 13:10:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568929  12 58442 47416761 A  ERRORB¢ff'
2018-11-20 13:10:27 [DEBUG   ] Finished updating ant stats in 0.597 seconds
2018-11-20 13:10:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:10:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:10:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:10:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:10:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:10:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:10:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:10:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:10:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:10:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:10:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:10:46 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:10:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:10:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:10:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:10:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:10:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:10:46 [DEBUG   ] Finished updating board flags in 1.753 seconds
2018-11-20 13:10:46 [DEBUG   ] Got command RPT from MCS: ref #568938
2018-11-20 13:10:46 [DEBUG   ] summary = ERROR
2018-11-20 13:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568938  13 58442 47446709 A  ERRORERROR'
2018-11-20 13:10:46 [DEBUG   ] Got command RPT from MCS: ref #568939
2018-11-20 13:10:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 13:10:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568939  98 58442 47446710 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 13:10:47 [DEBUG   ] Finished updating ant stats in 0.403 seconds
2018-11-20 13:11:08 [DEBUG   ] Finished updating ant stats in 1.397 seconds
2018-11-20 13:11:28 [DEBUG   ] Finished updating ant stats in 1.207 seconds
2018-11-20 13:11:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:11:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302373023568000000
2018-11-20 13:11:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:11:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:11:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:11:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:11:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302373023568000000
2018-11-20 13:11:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 13:11:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 13:11:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 13:11:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:11:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:11:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:11:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:11:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:11:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:11:46 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:11:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:11:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:11:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:11:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:11:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:11:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:11:46 [DEBUG   ] Finished updating board flags in 1.608 seconds
2018-11-20 13:11:48 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:12:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:12:28 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 13:12:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:12:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302373035328000000
2018-11-20 13:12:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:12:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:12:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:12:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:12:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302373035328000000
2018-11-20 13:12:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 13:12:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:12:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:12:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:12:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:12:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:12:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:12:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 13:12:46 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:12:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:12:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:12:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:12:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:12:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:12:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:12:46 [DEBUG   ] Finished updating board flags in 1.612 seconds
2018-11-20 13:12:48 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:13:08 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 13:13:28 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:13:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:13:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302373047088000000
2018-11-20 13:13:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:13:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:13:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:13:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:13:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302373047088000000
2018-11-20 13:13:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 13:13:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:13:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:13:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:13:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:13:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:13:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:13:46 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:13:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:13:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:13:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:13:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:13:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:13:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:13:46 [DEBUG   ] Finished updating board flags in 1.591 seconds
2018-11-20 13:13:48 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 13:14:08 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 13:14:28 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 13:14:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:14:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302373058848000000
2018-11-20 13:14:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:14:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:14:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:14:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:14:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302373058848000000
2018-11-20 13:14:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 13:14:45 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 13:14:45 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:14:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:14:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302373059044000000
2018-11-20 13:14:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:14:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:14:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:14:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:14:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302373059044000000
2018-11-20 13:14:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 13:14:46 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:14:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:14:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:14:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:14:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:14:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:14:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 13:14:46 [DEBUG   ] Finished updating board flags in 1.530 seconds
2018-11-20 13:14:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:15:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:15:16 [DEBUG   ] Got command RPT from MCS: ref #568959
2018-11-20 13:15:16 [DEBUG   ] summary = ERROR
2018-11-20 13:15:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568959  13 58442 47716037 A  ERRORERROR'
2018-11-20 13:15:16 [DEBUG   ] Got command RPT from MCS: ref #568960
2018-11-20 13:15:16 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 13:15:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568960  79 58442 47716241 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 13:15:16 [DEBUG   ] Got command RPT from MCS: ref #568961
2018-11-20 13:15:16 [DEBUG   ] summary = ERROR
2018-11-20 13:15:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568961  13 58442 47716445 A  ERRORERROR'
2018-11-20 13:15:16 [DEBUG   ] Got command RPT from MCS: ref #568962
2018-11-20 13:15:16 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 13:15:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568962  79 58442 47716649 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 13:15:16 [DEBUG   ] Got command RPT from MCS: ref #568963
2018-11-20 13:15:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 13:15:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568963  12 58442 47716841 A  ERRORB¡ff'
2018-11-20 13:15:28 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:15:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:15:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302373070608000000
2018-11-20 13:15:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:15:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:15:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:15:45 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:15:45 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302373070608000000
2018-11-20 13:15:45 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa23, chip 2
2018-11-20 13:15:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:15:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:15:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 1 with 302373070804000000
2018-11-20 13:15:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:15:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:15:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:15:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:15:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa34, chip 5 with 302373070804000000
2018-11-20 13:15:46 [WARNING ] MonitorFlags: VCCINT 0.981 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 13:15:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa25, chip 4
2018-11-20 13:15:46 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:15:46 [DEBUG   ] Got command RPT from MCS: ref #568964
2018-11-20 13:15:46 [DEBUG   ] summary = ERROR
2018-11-20 13:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568964  13 58442 47746713 A  ERRORERROR'
2018-11-20 13:15:46 [DEBUG   ] Got command RPT from MCS: ref #568965
2018-11-20 13:15:46 [DEBUG   ] info = SUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1
2018-11-20 13:15:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568965  79 58442 47746714 A  ERRORSUMMARY! 0x0A! DRX time not updating correctly - beam(s) #4, #3, #2, #1'
2018-11-20 13:15:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:15:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:15:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:15:46 [WARNING ] (13, 1, 'bot')
2018-11-20 13:15:46 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:15:46 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3, #2, #1
2018-11-20 13:15:46 [DEBUG   ] Finished updating board flags in 1.594 seconds
2018-11-20 13:15:48 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:16:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:16:28 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 13:16:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:16:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:16:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:16:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:16:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:16:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 13:16:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:16:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:16:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:16:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:16:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:16:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:16:47 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:16:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:16:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:16:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:16:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:16:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:16:47 [DEBUG   ] Finished updating board flags in 2.163 seconds
2018-11-20 13:16:48 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:17:08 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 13:17:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:17:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:17:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:17:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:17:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:17:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:17:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 13:17:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:17:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:17:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:17:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:17:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:17:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:17:47 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:17:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:17:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:17:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:17:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:17:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:17:47 [DEBUG   ] Finished updating board flags in 2.050 seconds
2018-11-20 13:17:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:18:08 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:18:28 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:18:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:18:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:18:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:18:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:18:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:18:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 13:18:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:18:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa26, chip 4
2018-11-20 13:18:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:18:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:18:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:18:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:18:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:18:47 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:18:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa7, chip 3
2018-11-20 13:18:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:18:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:18:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:18:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:18:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:18:47 [DEBUG   ] Finished updating board flags in 1.903 seconds
2018-11-20 13:18:48 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 13:19:08 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 13:19:28 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:19:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:19:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:19:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:19:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:19:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:19:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:19:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:19:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:19:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:19:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:19:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:19:47 [WARNING ] MonitorFlags: Temperature 80.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:19:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:19:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:19:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:19:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:19:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:19:47 [DEBUG   ] Finished updating board flags in 1.828 seconds
2018-11-20 13:19:48 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 13:20:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:20:16 [DEBUG   ] Got command RPT from MCS: ref #568991
2018-11-20 13:20:16 [DEBUG   ] summary = ERROR
2018-11-20 13:20:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568991  13 58442 48016121 A  ERRORERROR'
2018-11-20 13:20:16 [DEBUG   ] Got command RPT from MCS: ref #568992
2018-11-20 13:20:16 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 13:20:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568992  98 58442 48016325 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 13:20:16 [DEBUG   ] Got command RPT from MCS: ref #568993
2018-11-20 13:20:16 [DEBUG   ] summary = ERROR
2018-11-20 13:20:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568993  13 58442 48016521 A  ERRORERROR'
2018-11-20 13:20:16 [DEBUG   ] Got command RPT from MCS: ref #568995
2018-11-20 13:20:16 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 13:20:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568995  98 58442 48016725 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 13:20:16 [DEBUG   ] Got command RPT from MCS: ref #568997
2018-11-20 13:20:16 [DEBUG   ] GLOBAL_TEMP_MAX = exited with status True
2018-11-20 13:20:16 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   568997  12 58442 48016929 A  ERRORB ff'
2018-11-20 13:20:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:20:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:20:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:20:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:20:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:20:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:20:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 13:20:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa32, chip 1
2018-11-20 13:20:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:20:46 [DEBUG   ] Got command RPT from MCS: ref #569001
2018-11-20 13:20:46 [DEBUG   ] summary = ERROR
2018-11-20 13:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   569001  13 58442 48046709 A  ERRORERROR'
2018-11-20 13:20:46 [DEBUG   ] Got command RPT from MCS: ref #569002
2018-11-20 13:20:46 [DEBUG   ] info = T_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected
2018-11-20 13:20:46 [DEBUG   ] mcsSend - Sent to MCS 'MCSDP_RPT   569002  98 58442 48046710 A  ERRORT_NOM1 T_NOM4 T_NOM2 T_NOM3! 0x07! Beamformer calibration lost - MGT underflow(s) detected'
2018-11-20 13:20:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:20:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:20:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:20:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:20:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:20:47 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:20:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:20:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:20:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:20:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:20:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:20:47 [DEBUG   ] Finished updating board flags in 1.776 seconds
2018-11-20 13:20:48 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:21:08 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 13:21:28 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:21:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:21:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302373141364000000
2018-11-20 13:21:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:21:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:21:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:21:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:21:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302373141364000000
2018-11-20 13:21:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa18, chip 5
2018-11-20 13:21:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:21:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:21:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:21:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:21:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:21:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:21:47 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:21:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:21:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:21:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:21:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:21:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:21:47 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:21:47 [DEBUG   ] Finished updating board flags in 1.642 seconds
2018-11-20 13:21:48 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 13:22:08 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:22:28 [DEBUG   ] Finished updating ant stats in 1.204 seconds
2018-11-20 13:22:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:22:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302373153124000000
2018-11-20 13:22:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:22:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:22:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:22:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:22:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302373153124000000
2018-11-20 13:22:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:22:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:22:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:22:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:22:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:22:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:22:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa22, chip 2
2018-11-20 13:22:47 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:22:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:22:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:22:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:22:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:22:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:22:47 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:22:47 [DEBUG   ] Finished updating board flags in 1.663 seconds
2018-11-20 13:22:48 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:23:08 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 13:23:28 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:23:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:23:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302373164884000000
2018-11-20 13:23:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:23:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:23:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:23:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:23:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302373164884000000
2018-11-20 13:23:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:23:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:23:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:23:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:23:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:23:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:23:47 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa19, chip 5
2018-11-20 13:23:47 [WARNING ] MonitorFlags: Temperature 80.70 > 80 on hostname lwa21, chip 2
2018-11-20 13:23:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:23:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:23:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:23:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:23:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:23:47 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:23:47 [DEBUG   ] Finished updating board flags in 1.626 seconds
2018-11-20 13:23:48 [DEBUG   ] Finished updating ant stats in 1.203 seconds
2018-11-20 13:24:08 [DEBUG   ] Finished updating ant stats in 1.202 seconds
2018-11-20 13:24:28 [DEBUG   ] Finished updating ant stats in 1.201 seconds
2018-11-20 13:24:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 1 with 184959051
2018-11-20 13:24:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 1 with 302373176644000000
2018-11-20 13:24:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 2 with 184957055
2018-11-20 13:24:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 3 with 184950976
2018-11-20 13:24:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 4 with 184885461
2018-11-20 13:24:46 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa16, chip 5 with 184934238
2018-11-20 13:24:46 [WARNING ] MonitorFlags: DRX time not updating on hostname lwa16, chip 5 with 302373176644000000
2018-11-20 13:24:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa30, chip 4
2018-11-20 13:24:46 [WARNING ] MonitorFlags: VCCINT 0.984 out of range 0.987 to 1.04 on hostname lwa13, chip 5
2018-11-20 13:24:46 [WARNING ] MonitorFlags: MGT underflow on hostname lwa26, chip 1, bottom port
2018-11-20 13:24:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 1 with 183749872
2018-11-20 13:24:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 2 with 183749644
2018-11-20 13:24:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 3 with 183749654
2018-11-20 13:24:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 4 with 183749602
2018-11-20 13:24:47 [WARNING ] MonitorFlags: 1PPS unstable on hostname lwa34, chip 5 with 183749598
2018-11-20 13:24:47 [WARNING ] MonitorFlags: Temperature 81.20 > 80 on hostname lwa21, chip 2
2018-11-20 13:24:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:24:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:24:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:24:47 [WARNING ] (13, 1, 'bot')
2018-11-20 13:24:47 [CRITICAL] MGT underflow detected on beam(s) 1, 4, 2, 3
2018-11-20 13:24:47 [CRITICAL] DRX time not updating correctly on beam(s) #4, #3
2018-11-20 13:24:47 [DEBUG   ] Finished updating board flags in 1.644 seconds
2018-11-20 13:24:48 [DEBUG   ] Finished updating ant stats in 1.203 seconds
