08/12 16:38 UTC taking control of the array. Running TBN at 37.9 MHz 08/12 16:42 UTC ./generateFullStorageReport.sh DR1 DRSUs Attached: 2 DRSU selected: 2 DRSU1: /dev/md127 Barcode: ''S10TCC13S0008'' Total Storage: 9315.08 GB Remaining Storage: 8848.38 GB Usage: 5.00% File Count: 0 files DRSU2: /dev/md126 Barcode: ''S15TCV12S0001'' Total Storage: 13972.59 GB Remaining Storage: 6975.26 GB Usage: 50.00% File Count: 23 files =========== Storage report for DR2 =========== DR2 DRSUs Attached: 2 DRSU selected: 2 DRSU1: /dev/md127 Barcode: ''S10TCC13S0024'' Total Storage: 9315.08 GB Remaining Storage: 8848.38 GB Usage: 5.00% File Count: 0 files DRSU2: /dev/md126 Barcode: ''S15TCV13S0006'' Total Storage: 13972.59 GB Remaining Storage: 6540.05 GB Usage: 53.00% File Count: 30 files =========== Storage report for DR3 =========== DR3 DRSUs Attached: 2 DRSU selected: 2 DRSU1: /dev/md127 Barcode: ''S10TCC13S0022'' Total Storage: 9315.08 GB Remaining Storage: 8848.38 GB Usage: 5.00% File Count: 0 files DRSU2: /dev/md126 Barcode: ''S15TCV13S0005'' Total Storage: 13972.59 GB Remaining Storage: 6343.77 GB Usage: 54.00% File Count: 31 files =========== Storage report for DR4 =========== DR4 DRSUs Attached: 2 DRSU selected: 2 DRSU1: /dev/md127 Barcode: ''S10TCC13S0023'' Total Storage: 9315.08 GB Remaining Storage: 8848.38 GB Usage: 5.00% File Count: 0 files DRSU2: /dev/md126 Barcode: ''S15TCV13S0003'' Total Storage: 13972.59 GB Remaining Storage: 6797.83 GB Usage: 51.00% File Count: 26 files =========== Storage report for DR5 =========== DR5 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md0 Barcode: ''S10TCC13S0005'' Total Storage: 9315.08 GB Remaining Storage: 4086.98 GB Usage: 56.00% File Count: 185 files DRSU2: /dev/md127 Barcode: ''S10TCC13S0005'' Total Storage: 9315.08 GB Remaining Storage: 4086.98 GB Usage: 56.00% File Count: 185 files 08/13 03:00 UTC: ran LH009 0007 with full bandwidth (TBW) sdfs: LH009_130813_0300_0007_TBW.sdf file tags: 056517_000018036 - 056517_000019923 file sizes: 14G each duration: 8 hours 08/13 11:05: running TBN at 37.8 MHz 08/13 13:30 UTC: took a TBW capture 08/13 13:34 UTC: running runlatestcheck.sh and checkstatuschanges.sh 08/13 20:12 UTC: array is overheating. shutting down ASP and DP 08/13 21:33 UTC: 08 did an ASP INI and a DP INI, started TBN at 37.9 MHz 08/13 23:30 UTC: ran LS003 141 and 142 with split bandwidth on beams 3 and 4 sdfs: LS003_130813_2330_b3.sdf, LS003_130813_2330_b4.sdf file tags: 056517_000022685, 056517_000022686 file sizes: 266G, 266G duration: 1 hour 08/13 23:59 UTC: ran LS003 143-144 with split bandwidth one beams 1, 2 sdfs: LS003_130814_2359_b1.sdf, LS003_130813_2359_b2.sdf file tags: 056517_000022797, 056517_000022798 file sizes: 266G, 266G duration: 1 hour array is overheating. shutting down ASP and DP 08/14 00:57 UTC: ran LS003 145-146 with split bandwidth on beams 3, 4 sdfs: LS003_130814_0057_b3.sdf, LS003_130814_0057_b4.sdf file tags: 056518_000022994, 056518_000022994 file sizes: 266G duration: 1 hour 8/14 01:21 UTC: ran LS003 147-148 with split bandwidth on beams 1, 2 sdfs: LS003_130814_0121_b1.sdf, LS003_130814_0121_b2.sdf file tags: 056518_000023089, 056518_000023090 file sizes: 266G duration: 1 hour 8/14 02:03 UTC: ran LS003 149-150 with split bandwidth on beams 3, 4 sdfs: LS003_130814_0203_b3.sdf, LS003_130814_0203_b4.sdf file tags: 056518_000023235, 056518_000023236 file sizes: 266G duration: 1 hour 8/14 02:41 UTC: ran LS003 151-152 with split bandwidth on beams 1, 2 sdfs: LS003_130814_0241_b1.sdf, LS003_130814_0241_b2.sdf file tags: 056518_000023381, 056518_000023382 file sizes: 266G duration: 1 hour 08/14 03:42 UTC: restarting MS Executive to update submitted sdfs 08/14 04:03 UTC: ran LS003 155-156 with split bandwidth on beams 1, 2 sdfs: LS003_130814_0403_b1.sdf, LS003_130814_0403_b2.sdf file tags: 056518_000023679, 056518_000023678 file sizes: 266G duration: 1 hour 08/14 05:21 UTC: ran LS003 157-158 with split bandwidth on beams 1, 2 sdfs: LS003_130814_0528_b1.sdf, LS003_130814_0528_b2.sdf file tags: 056518_000023960, 056518_000023961 file sizes: 266G duration: 1 hour 08/14 07:21 UTC: ran LS003 161-164 with split bandwidth on beams 1-4 sdfs: LS003_130814_0721_b1.sdf, LS003_130814_0721_b2.sdf LS003_130814_0721_b3.sdf, LS003_130814_0721_b4.sdf file tags: 056518_000024378, 056518_000024379, 056518_000024380, 056518_000024381 file sizes: 266G duration: 1 hour 08/14 08:30 UTC: ran LS003 165-166 with split bandwidth on beams 3, 4 sdfs: LS003_130814_0830_b3.sdf, LS003_130814_0830_b4.sdf file tags: 056518_000024636, 056518_000024637 file sizes: 266G duration: 1 hour 08/14 09:41 UTC: MSexecutive crashed. DR4 reports not enough advanced notice 08:14 13:34 UTC: station is overheating, shutting off DP and ASP 08/14 15:30 UTC: bringing the station back online, running TBN at 37.8 MHz 08/14 17:14 UTC: doing a TBW analysis 08/15 00:23 UTC: ran LS003 187-188 with split bandwidth on beams 1, 2 sdfs: LS003_130815_0023_b1.sdf, LS003_130815_0023_b2.sdf file tags: 056519_000027751, 056519_000027752 file sizes: 266G duration: 1 hour 08/15 01:23 UTC: FPGA temperatures are getting high, keeping station active 08/15 01:44 UTC: ran LS003 189-190 with split bandwidth on beams 1, 2 sdfs: LS003_130815_0144_b1.sdf, LS003_130815_0144_b2.sdf file tags: 056519_000028033, 056519_00002804 file sizes: 266G duration: 1 hour 08/15 02:10 UTC: ran LS003 191-192 with split bandwidth on beams 3, 4 sdfs: LS003_130815_0210_b3.sdf, LS003_130815_0210_b4.sdf file tags: 056519_000028128, 056519_000028129 file sizes: 266G duration: 1 hour 08/15 03:03 UTC: ran LS003 193-194 with split bandwidth on beams 1, 2 sdfs: LS003_130815_0303_b1.sdf, LS003_130815_0303_b2.sdf file tags: 056519_000028325, 056519_000028326 file sizes: 266G duration: 1 hour 08/15 04:20 UTC: ran LS003 195-196 with split bandwidth on beams 3, 4 sdfs: LS003_130815_0420_b3.sdf, LS003_130815_0420_b4.sdf file tags: 056519_000028386, 056519_000028387 file sizes: 266G duration: 1 hour 08/15 04:35 UTC: ran COMJD 1074 and 1075 with split bandwidth on beams 3,4 sdfs: COMJD_130815_0435_B3.sdf, COMJD_130815_0435_B4.sdf file tags: 056519_000028656, 056519_000028651 file sizes: 55G duration: 2 hours 08/15 04:38 UTC: ran LS003 197-198 with split bandwidth on beams 1, 2 sdfs: LS003_130815_0438_b1.sdf, LS003_130815_0438_b2.sdf file tags: 056519_000028678, 056519_000028679 file sizes: 266G duration: 1 hour 08/15 06:16 UTC: ran LS003 201-202 with split bandwidth on beams 1, 2 sdfs: LS003_130815_0616_b1.sdf, LS003_130815_0616_b2.sdf file tags: 056519_000029028, 056519_000029029 file sizes: 266G duration: 1 hour 08/15 07:17 UTC: ran LS003 203-206 with split bandwidth on beams 1-4 sdfs: LS003_130815_0717_b1.sdf, LS003_130815_0717_b2.sdf LS003_130815_0717_b3.sdf, LS003_130815_0717_b4.sdf file tags: 056519_000029242, 056519_000029243, 056519_000029244, 056519_000029245 file sizes: 266G duration: 1 hour 08/15 08:47 UTC: ran LS003 207-208 with split bandwidth on beams 3, 4 sdfs: LS003_130815_0847_b3.sdf, LS003_130815_0847_b4.sdf file tags: 056519_000029568, 056519_000029569 file sizes: 266G duration: 1 hour 08/15 08:52 UTC: ran LS003 209-210 with split bandwidth on beams 1, 2 sdfs: LS003_130815_0852_b1.sdf, LS003_130815_0852_b2.sdf file tags: 056519_000029595, 056519_000029596 file sizes: 266G duration: 1 hour 08/15 09:10 UTC: FPGA temps have decreased below warning levels 08/15 10:15 UTC: ran LS003 211-212 with split bandwidth on beams 3, 4 sdfs: LS003_130815_1015_b3.sdf, LS003_130815_1015_b4.sdf file tags: 056519_000029877, 056519_000029878 file sizes: 266G duration: 1 hour 08/15 10:43 UTC: ran LS003 213-214 with split bandwidth on beams 1, 2 sdfs: LS003_130815_1043_b1.sdf, LS003_130815_1043_b2.sdf file tags: 056519_000029989, 056519_000029990 file sizes: 266G duration: 1 hour 08/15 11:50 UTC: running DP_ INI and starting TBN at 37.8 MHz 08/15 12:02 UTC: stopping the beams 08/15 14:49 UTC: doing a TBW analysis 08/15 15:02 UTC: running TBN at 37.9 MHz 08/15 23:23 UTC: ran LS003 241-242 with split bandwidth on beams 1,2 sdfs: LS003_130815_2323_b1.sdf, LS003_130815_2323_b2.sdf file tags: 056519_000032850, 056519_000032851 file sizes: 266G duration: 1 hour 08/16 02:36 UTC: ran LS003 215-216 with split bandwidth on beams 3,4 sdfs: LS003_130816_0236_b3.sdf, LS003_130816_0236_b4.sdf file tags: 056520_000033524, 056520_000033523 file sizes: 266G duration: 1 hour 08/16 03:26 UTC: ran LS003 217-218 with split bandwidth on beams 1,2 sdfs: LS003_130816_0353_b1.sdf, LS003_130816_0353_b2.sdf file tags: 056520_000033704, 056520_000033703 file sizes: 266G duration: 1 hour 08/16 03:53 UTC: ran LS003 219-220 with split bandwidth on beams 3,4 sdfs: LS003_130816_0353_b3.sdf, LS003_130816_0353_b4.sdf file tags: 056520_000033799, 056520_000033798 file sizes: 266G duration: 1 hour 08/16 06:36 UTC: ran LS003 221-222 with split bandwidth on beams 1,2 sdfs: LS003_130816_0636_b1.sdf, LS003_130816_0636_b2.sdf file tags: 056520_000034370, 056520_000034369 file sizes: 266G duration: 1 hour 08/16 06:43 UTC: ran LS003 223-224 with split bandwidth on beams 3,4 sdfs: LS003_130816_0643_b1.sdf, LS003_130816_0643_b2.sdf file tags: 056520_000034397, 056520_000034396 file sizes: 266G duration: 1 hour 08/16 07:55 UTC: ran LS003 225-226 with split bandwidth on beams 1,2 sdfs: LS003_130816_2323_b1.sdf, LS003_130816_2323_b2.sdf file tags: 056520_000034645, 056520_000034644 file sizes: 266G duration: 1 hour 08/16 08:07 UTC: ran LS003 227-228 with split bandwidth on beams 3,4 sdfs: LS003_130816_0807_b3.sdf, LS003_130816_0807_b4.sdf file tags: 056520_000034706, 056520_000034705 file sizes: 266G duration: 1 hour 08/16 09:15 UTC: ran LS003 229-230 with split bandwidth on beams 1,2 sdfs: LS003_130816_0915_b1.sdf, LS003_130816_0915_b2.sdf file tags: 056520_000034937, 056520_000034936 file sizes: 266G duration: 1 hour 08/16 09:43 UTC: ran LS003 169-170 with split bandwidth on beams 3,4 sdfs: LS003_130816_0943_b3.sdf, LS003_130816_0943_b4.sdf file tags: 056520_000035048, 056520_000035049 file sizes: 266G duration: 1 hour 08/16 10:54 UTC: ran LS003 173-174 with split bandwidth on beams 1,2 sdfs: LS003_130816_1054_b1.sdf, LS003_130816_1054_b2.sdf file tags: 056520_000035296, 056520_000035297 file sizes: 266G duration: 1 hour 08/16 11:36 UTC: ran LS003 175-176 with split bandwidth on beams 3,4 sdfs: LS003_130816_1136_b3.sdf, LS003_130816_1136_b4.sdf file tags: 056520_000035459, 056520_000035460 file sizes: 266G duration: 1 hour 08/16 12:15 UTC: ran LS003 177-178 with split bandwidth on beams 1,2 sdfs: LS003_130816_1215_b1.sdf, LS003_130816_1215_b2.sdf file tags: 056520_000035588, 056520_000035589 file sizes: 266G duration: 1 hour 08/16 12:54 UTC: ran LS003 179-180 with split bandwidth on beams 3,4 sdfs: LS003_130816_1254_b3.sdf, LS003_130816_1254_b4.sdf file tags: 056520_000035734, 056520_000035735 file sizes: 266G duration: 1 hour problems: station had to be shut down mid-run due to overheating. 08/16 13:15 UTC: station is overheating, shutting off ASP and DP 08/16 15:11 UTC: booting up ASP and DP, bringint the station back online 08/16 15:13 UTC: doing a TBW analysis 08/16 15:34 UTC: starting TBN at 37.9 MHZ 08/16 18:58 UTC: shutting down station so that it is thawed for LW001 run 08/16 21:07 UTC: bringing station back online; running TBN at 37.9 MHz 08/16 22:00 UTC: ran DW001 001-004 with split bandwidth on beams 1,4 sdfs: DW001_130816_2200_0001_B1.sdf, DW001_130816_2200_0002_B2.sdf, DW001_130816_2200_0003_B3.sdf, DW001_130816_2200_0004_B4.sdf file tags: 056520_000037980, 056520_000037981, 056520_000037982, 056520_000037983 file sizes: 527G duration: 2 hours 08/17 00:00 UTC: ran DW001 005-008 with split bandwidth on beams 1,2 sdfs: DW001_130816_2400_0005_B1.sdf, DW001_130816_2400_0006_B2.sdf, DW001_130816_2400_0007_B3.sdf, DW001_130816_2400_0008_B4.sdf file tags: 056520_000039062, 056520_0000390623, 056520_000039064, 056520_000039065 file sizes: 531G duration: 2 hours 08/17 03:20 UTC: ran LS003 153-154 with split bandwidth on beams 1,2 sdfs: LS003_130817_0320_b1.sdf, LS003_130817_0320_b2.sdf file tags: 056521_000040648, 056521_000040649 file sizes: 266G duration: 1 hour 08/17 03:51 UTC: ran LS003 231-232 with split bandwidth on beams 3,4 sdfs: LS003_130817_0351_b3.sdf, LS003_130817_0351_b4.sdf file tags: 056521_000040777, 056521_000040778 file sizes: 266G duration: 1 hour 08/17 05:51 UTC: ran LS003 159-160 with split bandwidth on beams 3,4 sdfs: LS003_130817_0551_b3.sdf, LS003_130817_0551_b4.sdf file tags: 056521_000041195, 056521_000041196 file sizes: 266G duration: 1 hour 08/17 06:59 UTC: ran LS003 233-234 with split bandwidth on beams 1,2 sdfs: LS003_130817_0659_b1.sdf, LS003_130817_0659_b2.sdf file tags: 056521_000041426, 056521_000041427 file sizes: 266G duration: 1 hour 08/17 07:25 UTC: ran LS003 235-236 with split bandwidth on beams 3,4 sdfs: LS003_130817_0725_b3.sdf, LS003_130817_0725_b4.sdf file tags: 056521_000041521, 056521_000041522 file sizes: 266G duration: 1 hour 08/17 10:02 UTC: ran LS003 171-172 with split bandwidth on beams 1,2 sdfs: LS003_130817_1002_b1.sdf, LS003_130817_1002_b2.sdf file tags: 056521_000042075, 056521_000042075 file sizes: 266G duration: 1 hour 08/17 12:50 UTC: ran LS003 179-180 with split bandwidth on beams 3,4 sdfs: LS003_130817_1250_b3.sdf, LS003_130817_1250_b4.sdf file tags: 056521_000042646, 056521_000042647 file sizes: 266G duration: 1 hour 08/17 13:29 UTC: ran LS003 181-182 with split bandwidth on beams 1,2 sdfs: LS003_130817_1329_b1.sdf, LS003_130817_1329_b2.sdf file tags: 056521_000042792, 056521_000042793 file sizes: 266G duration: 1 hour 08/17 14:09 UTC: ran LS003 183-184 with split bandwidth on beams 3,4 sdfs: LS003_130817_1409_b3.sdf, LS003_130817_1409_b4.sdf file tags: 056521_000042938, 056521_000042939 file sizes: 266G duration: 1 hour 08/17 14:50 UTC: ran LS003 185-186 with split bandwidth on beams 1,2 sdfs: LS003_130817_1450_b1.sdf, LS003_130817_1450_b2.sdf file tags: 056521_000043084, 056521_000043085 file sizes: 266G duration: 1 hour 08/17 16:11 UTC: doing a TBW analysis 08/17 16:22 UTC: running TBN at 37.8 MHz 08/18 20:22 UTC: station is overheating. Turning off DP and ASP 08/18 22:35 UTC: bringing station back online; lighting in vicinity, starting lightning mode 08/19 00:37 UTC: lightning condition cleared, running TBN at 37.9 MHZ station is overheating. Turning off DP and ASP 08/18 03:00 UTC: ran LH009 005 with full bandwidth (TBW) sdfs: LH009_130818_0300_TBW.sdf file tags: 056522_000046036 - 056522_000047892 file sizes: 8.9G duration: 8 hours issues: several shelter brown-outs occured during observation. Although files were written for each observation, some were only 4kb in size. 08/18 11:10 UTC: running TBN at 37.9 MHz 08/18 17:56 UTC: doing a TBW analysis 08/18 18:02 UTC: running TBN at 37.9 MHz 08/18 22:04 UTC: PASI stopped updating LWAtv. running pasicmd restart 08/19 04:26 UTC: ran LS003 199-200 with split bandwidth on beams 3,4 sdfs: LS003_130819_0426_b3.sdf, LS003_130819_0426_b4.sdf file tags: 056523_000051692, 056523_000051693 file sizes: 266G duration: 1 hour 08/19 07:01 UTC: ran LS003 237-240 with split bandwidth on beams 1-4 sdfs: LS003_130819_0701_237_B1.sdf, LS003_130819_0701_238_b2.sdf, LS003_130819_0701_239_B3.sdf, LS003_130819_0701_240_b4.sdf file tags: 056523_000052310, 056523_000052311, 056523_000052312 file sizes: 266G duration: 1 hour issues: beam 1 did not record any data due to an operator scheduling error 08/19 08:10 UTC: running TBN at 37.9 MHz