G.B. Taylor taking over Downtime for the week=6 hours not counting various glitches on spectrometer observations. op1@mcs-tp:~/MCS/sch/operatorScrips$ ./generateFullStorageReport.sh 1845 UTC set to full bandwidth configuration 1856 UTC took TBW capture, looks good 1910 UTC started 4 beam solar observation to capture flares, 1024 ch spect. LW001_121126_1910_0026_B1.SDF OK LW001_121126_1910_0027_B2.SDF DR became unresponsive LW001_121126_1910_0028_B3.SDF OK LW001_121126_1910_0029_B4.SDF DR became unresponsive 121126 19:09:55 56257 68995170 T 12380 3 DR1 SPC 056257_000012380| 121126 19:09:55 56257 68995170 T 12381 3 DR2 SPC 056257_000012381| 121126 19:09:55 56257 68995106 T 12382 3 DR3 SPC 056257_000012382| 121126 19:09:55 56257 68995170 T 12383 3 DR4 SPC 056257_000012383| storage report: =========== Storage report for DR1 =========== DR1 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md127 Barcode: 'S10TCC13S0016' Total Storage: 9315.08 GB Remaining Storage: 5426.79 GB Usage: 41.00% File Count: 17 files DRSU2: /dev/md126 Barcode: ''S15TCV12S0002'' Total Storage: 13972.59 GB Remaining Storage: 13272.59 GB Usage: 5.00% File Count: 0 files =========== Storage report for DR2 =========== DR2 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md127 Barcode: ''S10TCC13S0008'' Total Storage: 9315.08 GB Remaining Storage: 5059.64 GB Usage: 45.00% File Count: 9 files DRSU2: /dev/md126 Barcode: ''S15TCV13S0001'' Total Storage: 13972.59 GB Remaining Storage: 13272.59 GB Usage: 5.00% File Count: 0 files =========== Storage report for DR3 =========== DR3 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md127 Barcode: 'S10TCC13S0024' Total Storage: 9315.08 GB Remaining Storage: 7068.24 GB Usage: 24.00% File Count: 3 files DRSU2: /dev/md126 Barcode: ''S15TCV12S0001'' Total Storage: 13972.59 GB Remaining Storage: 11379.84 GB Usage: 18.00% File Count: 4 files =========== Storage report for DR4 =========== DR4 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md127 Barcode: ''S10TCC13S0005'' Total Storage: 9315.08 GB Remaining Storage: 4187.47 GB Usage: 55.00% File Count: 13 files DRSU2: /dev/md126 Barcode: ''S15TCV13S0002'' Total Storage: 13972.59 GB Remaining Storage: 13272.59 GB Usage: 5.00% File Count: 0 files =========== Storage report for DR5 =========== DR5 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md127 Barcode: ''S15TCV13S0003'' Total Storage: 13972.59 GB Remaining Storage: 13272.59 GB Usage: 5.00% File Count: 0 files DRSU2: /dev/md126 Barcode: ''S10TCC13S0014'' Total Storage: 9315.08 GB Remaining Storage: 6947.93 GB Usage: 25.00% File Count: 7 files 2350 PASI is down, tried restarting PASI with no luck. Contacted Jake. Still in full bandwidth filter config Jupiter observations: LH004_121127_0300_0033_B2.SDF LH004_121127_0300_0034_B3.SDF 121127 02:59:55 56258 10795137 T 14637 3 DR2 REC 056258_000014637| 121127 02:59:55 56258 10795206 T 14638 3 DR3 REC 056258_000014638| spectrometer set full bandwidth filter config LW001_121127_1810_0030_B1.SDF fail LW001_121127_1810_0031_B2.SDF fail LW001_121127_1810_0032_B3.SDF fail LW001_121127_1810_0033_B4.SDF fail 121127 18:09:55 56258 65395198 T 18061 3 DR1 SPC 056258_000018061| 121127 18:09:55 56258 65395198 T 18062 3 DR2 SPC 056258_000018062| 121127 18:09:55 56258 65395113 T 18063 3 DR3 SPC 056258_000018063| 121127 18:09:55 56258 65395198 T 18064 3 DR4 SPC 056258_000018064| DP failed to INI so these all got rejected, deleted LW001_121127_2000_0034_B1.SDF fail LW001_121127_2000_0035_B2.SDF fail LW001_121127_2000_0036_B3.SDF fail LW001_121127_2000_0037_B4.SDF fail 121127 19:59:55 56258 71995182 T 18708 3 DR1 SPC 056258_000018708| 121127 19:59:55 56258 71995183 T 18710 3 DR2 SPC 056258_000018710| 121127 19:59:55 56258 71995128 T 18709 3 DR3 SPC 056258_000018709| 121127 19:59:55 56258 71995182 T 18711 3 DR4 SPC 056258_000018711| DRs all hung, files deleted 05:35 UTC stop beams and start TBN ASP was in a funny state. Several bad antennas and sky didn't look quite right. Tried INI twice and that just made it worse. (half the antennas went bad). So I did ASP SHT waited a few minutes, then turned ASP back on, did an INI, set filters, FPW and all was good. 1500 UTC LW001 observations, but they failed at 0800 and again at 0900 cleaned up, reported failures to Chris. Full bandwidth filter is set Scheduled: 0751 UTC = 0051 UTC op1@mcs-exec:~/MCS/sch$ at 00:21 11/29/12 -f INIdp.sh warning: commands will be executed using /bin/sh job 321 at Thu Nov 29 00:21:00 2012 LE002_121129_0741_0151_B_GP.sdf LE002_121129_0741_0152_B_GP.sdf LE002_121129_0741_0153_B_GP.sdf LE002_121129_0741_0154_B_GP.sdf 121129 07:40:55 56260 27655249 T 3199 3 DR1 REC 056260_000003199| 255 121129 07:40:55 56260 27655249 T 3196 3 DR2 REC 056260_000003196| 252 121129 07:40:55 56260 27655249 T 3197 3 DR3 REC 056260_000003197| no 121129 07:40:55 56260 27655150 T 3198 3 DR4 REC 056260_000003198| 251 no data recorded on drive 3 - bitten by upgrade op1@mcs-exec:~/MCS/exec$ at 02:00 11/29/12 -f stopandTBN.sh warning: commands will be executed using /bin/sh job 322 at Thu Nov 29 02:00:00 2012 PASI running for 5.5 hours op1@mcs-exec:~/MCS/sch$ at 07:30 11/29/30 -f INIdp.sh warning: commands will be executed using /bin/sh job 323 at Fri Nov 29 07:30:00 2030 start the beams running beforehand: LW001_121129_1450_1038_B1.SDF raw data LW001_121129_1450_1039_B2.SDF LW001_121129_1450_1040_B3.SDF LW001_121129_1450_1041_B4.SDF 121129 14:49:55 56260 53395257 T 5675 3 DR1 REC 056260_000005675| 121129 14:49:55 56260 53395257 T 5676 3 DR2 REC 056260_000005676| 121129 14:49:55 56260 53395140 T 5677 3 DR3 REC 056260_000005677| 121129 14:49:55 56260 53395257 T 5678 3 DR4 REC 056260_000005678| LW001_121129_1500_0038_B1.SDF spectrometer 1024 IQUV LW001_121129_1500_0039_B2.SDF LW001_121129_1500_0040_B3.SDF LW001_121129_1500_0041_B4.SDF 121129 14:59:55 56260 53995244 T 5730 3 DR1 SPC 056260_000005730| 121129 14:59:55 56260 53995244 T 5731 3 DR2 SPC 056260_000005731| 121129 14:59:55 56260 53995169 T 5732 3 DR3 SPC 056260_000005732| 121129 14:59:55 56260 53995244 T 5733 3 DR4 SPC 056260_000005733| 121129_2310_COMFS_0137_S1.sdf 121129_2310_COMFS_0138_S2.sdf 121129_2310_COMFS_0139_S3.sdf 121129_2310_COMFS_0140_S4.sdf 121129 23:09:55 56260 83395235 T 8404 3 DR1 SPC 056260_000008404| 121129 23:09:55 56260 83395235 T 8407 3 DR2 SPC 056260_000008407| 121129 23:09:55 56260 83395141 T 8406 3 DR3 SPC 056260_000008406| 121129 23:09:55 56260 83395235 T 8405 3 DR4 SPC 056260_000008405| choked near the end of the file with an "insufficient notice of BAM"... this crashed exec so no tarball was produced but the data might be ok 121129_2345_COMFS_0141_S1.sdf 121129_2345_COMFS_0142_S2.sdf 121129 23:44:55 56260 85495205 T 8681 3 DR1 SPC 056260_000008681| 121129 23:44:55 56260 85495112 T 8682 3 DR2 SPC 056260_000008682| 121130_0145_COMFS_0143_S1.sdf 121130_0145_COMFS_0144_S2.sdf 121130_0145_COMFS_0145_S3.sdf 121130_0145_COMFS_0146_S4.sdf 121130 01:44:55 56261 6295194 T 9130 3 DR1 SPC 056261_000009130| 121130 01:44:55 56261 6295269 T 9131 3 DR2 SPC 056261_000009131| 121130 01:44:55 56261 6295269 T 9132 3 DR3 SPC 056261_000009132| 121130 01:44:55 56261 6295269 T 9133 3 DR4 SPC 056261_000009133| 121130_0220_COMFS_0147_S3.sdf 121130_0220_COMFS_0148_S4.sdf 121130 02:19:55 56261 8395110 T 9355 3 DR3 SPC 056261_000009355| 121130 02:19:55 56261 8395193 T 9356 3 DR4 SPC 056261_000009356| COMFS_121130_0655_0149_S1.sdf COMFS_121130_0655_0150_S2.sdf 121130 06:54:55 56261 24895166 T 10339 3 DR1 SPC 056261_000010339| 121130 06:54:55 56261 24895166 T 10340 3 DR2 SPC 056261_000010340| COMFS_121130_0732_0129_S1.sdf COMFS_121130_0732_0130_S2.sdf COMFS_121130_0732_0131_S3.sdf COMFS_121130_0732_0132_S4.sdf 121130 07:32:23 56261 27143778 T 10558 3 DR1 SPC 056261_000010558| 121130 07:32:23 56261 27143861 T 10559 3 DR2 SPC 056261_000010559| 121130 07:32:23 56261 27143861 T 10560 3 DR3 SPC 056261_000010560| 121130 07:32:23 56261 27143861 T 10561 3 DR4 SPC 056261_000010561| COMFS_121130_0807_0133_S1.sdf COMFS_121130_0807_0134_S2.sdf COMFS_121130_0807_0135_S3.sdf COMFS_121130_0807_0136_S4.sdf 121130 08:07:23 56261 29243769 T 10971 3 DR1 SPC 056261_000010971| 121130 08:07:23 56261 29243856 T 10972 3 DR2 SPC 056261_000010972| 121130 08:07:23 56261 29243856 T 10974 3 DR3 SPC 056261_000010974| 121130 08:07:23 56261 29243856 T 10973 3 DR4 SPC 056261_000010973| 0915 UTC stop beams op1@mcs-exec:~/MCS/exec$ at 02:15 11/30/12 -f stopandTBN.sh warning: commands will be executed using /bin/sh job 325 at Fri Nov 30 02:15:00 2012 LH003_121130_1245_0010_TBN.SDF 121130 12:44:55 56261 45895097 T 12255 3 DR5 REC 056261_000012255| copied to Hercules for Joe Helmboldt 1325 UTC start beams back up op1@mcs-exec:~/MCS/sch$ at 06:25 11/30/12 -f INIdp.sh warning: commands will be executed using /bin/sh job 326 at Fri Nov 30 06:25:00 2012 1345 one minute dummy scans pointing at NCP DUMMY_121130_1345_1001_B1.SDF DUMMY_121130_1345_1002_B2.SDF DUMMY_121130_1345_1003_B3.SDF DUMMY_121130_1345_1004_B4.SDF 121130 13:44:55 56261 49495230 T 12464 3 DR1 REC 056261_000012464| 121130 13:44:55 56261 49495230 T 12465 3 DR2 REC 056261_000012465| 121130 13:44:55 56261 49495146 T 12466 3 DR3 REC 056261_000012466| 121130 13:44:55 56261 49495230 T 12467 3 DR4 REC 056261_000012467| 121130_1355_COMFS_0155_S3.sdf 121130_1355_COMFS_0156_S4.sdf 121130 13:54:55 56261 50095161 T 12514 3 DR3 SPC 056261_000012514| 121130 13:54:55 56261 50095161 T 12515 3 DR4 SPC 056261_000012515| 121130_1430_COMFS_0151_S1.sdf 121130_1430_COMFS_0152_S2.sdf 121130_1430_COMFS_0153_S3.sdf 121130_1430_COMFS_0154_S4.sdf 121130 14:29:55 56261 52195149 T 12715 3 DR1 SPC 056261_000012715| 121130 14:29:55 56261 52195237 T 12717 3 DR2 SPC 056261_000012717| 121130 14:29:55 56261 52195237 T 12716 3 DR3 SPC 056261_000012716| 121130 14:29:55 56261 52195237 T 12718 3 DR4 SPC 056261_000012718| 1515 UTC = 8:15 MT stopandTBN op1@mcs-exec:~/MCS/exec$ at 08:15 11/30/12 -f stopandTBN.sh warning: commands will be executed using /bin/sh job 327 at Fri Nov 30 08:15:00 2012 COMFS_121130_2306_0157_S1.sdf COMFS_121130_2306_0158_S2.sdf COMFS_121130_2306_0159_S3.sdf COMFS_121130_2306_0160_S4.sdf 121130 23:05:59 56261 83159265 T 14762 3 DR1 SPC 056261_000014762| 121130 23:05:59 56261 83159324 T 14765 3 DR2 SPC 056261_000014765| 121130 23:05:59 56261 83159264 T 14764 3 DR3 SPC 056261_000014764| 121130 23:05:59 56261 83159324 T 14763 3 DR4 SPC 056261_000014763| COMFS_121130_2341_0161_S1.sdf COMFS_121130_2341_0162_S2.sdf 121130 23:40:59 56261 85259221 T 15063 3 DR1 SPC 056261_000015063| 121130 23:40:59 56261 85259295 T 15064 3 DR2 SPC 056261_000015064| COMFS_121201_0141_0163_S1.sdf COMFS_121201_0141_0164_S2.sdf COMFS_121201_0141_0165_S3.sdf COMFS_121201_0141_0166_S4.sdf 121201 01:40:59 56262 6059240 T 15512 3 DR1 SPC 056262_000015512| 121201 01:40:59 56262 6059347 T 15513 3 DR2 SPC 056262_000015513| 121201 01:40:59 56262 6059240 T 15514 3 DR3 SPC 056262_000015514| 121201 01:40:59 56262 6059347 T 15511 3 DR4 SPC 056262_000015511| COMFS_121201_0216_0167_S3.sdf COMFS_121201_0216_0168_S4.sdf 121201 02:15:59 56262 8159202 T 15732 3 DR3 SPC 056262_000015732| 121201 02:15:59 56262 8159272 T 15733 3 DR4 SPC 056262_000015733| COMFS_121201_0651_0175_S1.sdf COMFS_121201_0651_0176_S2.sdf 121201 06:50:59 56262 24659249 T 16717 3 DR1 SPC 056262_000016717| 121201 06:50:59 56262 24659308 T 16718 3 DR2 SPC 056262_000016718| COMFS_121201_0728_0177_S1.sdf COMFS_121201_0728_0178_S2.sdf COMFS_121201_0728_0179_S3.sdf COMFS_121201_0728_0180_S4.sdf 121201 07:28:27 56262 26907885 T 16919 3 DR1 SPC 056262_000016919| 121201 07:28:27 56262 26907957 T 16920 3 DR2 SPC 056262_000016920| 121201 07:28:27 56262 26907957 T 16921 3 DR3 SPC 056262_000016921| 121201 07:28:27 56262 26907957 T 16922 3 DR4 SPC 056262_000016922| LH004_121201_0815_0037_B1.SDF LH004_121201_0815_0035_B2.SDF LH004_121201_0815_0036_B3.SDF LH004_121201_0815_0038_B4.SDF 121201 08:14:55 56262 29695135 T 17385 3 DR1 SPC 056262_000017385| 121201 08:14:55 56262 29695200 T 17383 3 DR2 REC 056262_000017383| 121201 08:14:55 56262 29695200 T 17384 3 DR3 REC 056262_000017384| 121201 08:14:55 56262 29695135 T 17386 3 DR4 SPC 056262_000017386| COMFS_121201_1351_0169_S3.sdf COMFS_121201_1351_0170_S4.sdf 121201 13:50:59 56262 49859262 T 20288 3 DR3 SPC 056262_000020288| 121201 13:50:59 56262 49859198 T 20289 3 DR4 SPC 056262_000020289| COMFS_121201_1426_0171_S1.sdf COMFS_121201_1426_0172_S2.sdf COMFS_121201_1426_0173_S3.sdf COMFS_121201_1426_0174_S4.sdf 121201 14:25:59 56262 51959228 T 20489 3 DR1 SPC 056262_000020489| 121201 14:25:59 56262 51959300 T 20490 3 DR2 SPC 056262_000020490| 121201 14:25:59 56262 51959228 T 20491 3 DR3 SPC 056262_000020491| 121201 14:25:59 56262 51959300 T 20492 3 DR4 SPC 056262_000020492| COMAG_121201_2130_005_B2.sdf COMAG_121201_2130_006_B3.sdf 121201 21:29:55 56262 77395124 T 22142 3 DR3 REC 056262_000022142| 121201 21:29:55 56262 77395191 T 22141 3 DR2 REC 056262_000022141| copied to Hercules for Alex Garcia and Kevin Stovall COMFS_121201_2302_0175_S1.sdf COMFS_121201_2302_0176_S2.sdf COMFS_121201_2302_0177_S3.sdf COMFS_121201_2302_0178_S4.sdf 121201 23:02:03 56262 82923300 T 22616 3 DR1 SPC 056262_000022616| 121201 23:02:03 56262 82923379 T 22619 3 DR2 SPC 056262_000022619| 121201 23:02:03 56262 82923300 T 22618 3 DR3 SPC 056262_000022618| 121201 23:02:03 56262 82923379 T 22617 3 DR4 SPC 056262_000022617| COMFS_121201_2337_0179_S1.sdf COMFS_121201_2337_0180_S2.sdf 121201 23:37:03 56262 85023296 T 22917 3 DR1 SPC 056262_000022917| 121201 23:37:03 56262 85023366 T 22918 3 DR2 SPC 056262_000022918| running PASI 0815 brown out, lost ASP and DP 1430 station back to life brown out, lost ASP and DP 2100 UTC INIdp.sh to stop TBN op1@mcs-exec:~/MCS/sch$ at 21:00 12/03/12 -f INIdp.sh warning: commands will be executed using /bin/sh job 331 at Mon Dec 3 21:00:00 2012 scheduled: LH004_121204_0430_1039_B1.SDF LH004_121204_0430_0039_B2.SDF LH004_121204_0430_0040_B3.SDF LH004_121204_0430_1040_B4.SDF 121204 04:29:55 56265 16195126 T 33801 3 DR1 SPC 056265_000033801| no 121204 04:29:55 56265 16195199 T 33802 3 DR2 REC 056265_000033802| no 121204 04:29:55 56265 16195199 T 33803 3 DR3 REC 056265_000033803| no 121204 04:29:55 56265 16195199 T 33800 3 DR4 SPC 056265_000033800| no Due to bug in DROSv2 the data did not record metafiles deleted LE002_121204_0720_0155_B_GP.sdf LE002_121204_0720_0156_B_GP.sdf LE002_121204_0720_0157_B_GP.sdf LE002_121204_0720_0158_B_GP.sdf 121204 07:21:15 56265 26475686 T 35149 3 DR1 REC 056265_000035149| 121204 07:21:15 56265 26475686 T 35146 3 DR2 REC 056265_000035146| 121204 07:21:15 56265 26475625 T 35147 3 DR3 REC 056265_000035147| 121204 07:21:15 56265 26475686 T 35148 3 DR4 REC 056265_000035148| MCS executive died about 20 minutes into run due to unanticipated BAM no metadata produced stop beams and launch TBN op1@mcs-exec:~/MCS/exec$ at 02:30 12/04/12 -f stopandTBN.sh warning: commands will be executed using /bin/sh job 333 at Tue Dec 4 02:30:00 2012 =========== Storage report for DR1 =========== DR1 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md127 Barcode: 'S10TCC13S0016' Total Storage: 9315.08 GB Remaining Storage: 4840.82 GB Usage: 48.00% File Count: 22 files DRSU2: /dev/md126 Barcode: ''S15TCV12S0002'' Total Storage: 13972.59 GB Remaining Storage: 13272.59 GB Usage: 5.00% File Count: 0 files =========== Storage report for DR2 =========== DR2 DRSUs Attached: 2 DRSU selected: 1 [6/6077] LWA_dbm_fetch() failed; label=<DRSU-BARCODE> DRSU1: /dev/md127 Barcode: ''S10TCC13S0008'' Total Storage: 9315.08 GB Remaining Storage: 2467.00 GB Usage: 73.00% File Count: 17 files DRSU2: /dev/md126 Barcode: ''S15TCV13S0001'' Total Storage: 13972.59 GB Remaining Storage: 13272.59 GB Usage: 5.00% File Count: 0 files =========== Storage report for DR3 =========== DR3 DRSUs Attached: 2 DRSU selected: 1 DRSU1: /dev/md127 Barcode: 'S10TCC13S0024' Total Storage: 9315.08 GB Remaining Storage: 4758.25 GB Usage: 48.00% File Count: 11 files DRSU2: /dev/md126 Barcode: ''S15TCV12S0001'' Total Storage: 13972.59 GB Remaining Storage: 11379.84 GB Usage: 18.00% File Count: 4 files =========== Storage report for DR4 =========== DR4 DRSUs Attached: 2 DRSU selected: 1 [6/6849] LWA_dbm_fetch() failed; label=<DRSU-BARCODE> DRSU1: /dev/md127 Barcode: ''S10TCC13S0005'' Total Storage: 9315.08 GB Remaining Storage: 3623.50 GB Usage: 61.00% File Count: 17 files DRSU2: /dev/md126 Barcode: ''S15TCV13S0002'' Total Storage: 13972.59 GB Remaining Storage: 13272.59 GB Usage: 5.00% File Count: 0 files =========== Storage report for DR5 =========== DR5 DRSUs Attached: 2 DRSU selected: 1 [6/6965] LWA_dbm_fetch() failed; label=<DRSU-BARCODE> DRSU1: /dev/md127 Barcode: ''S15TCV13S0006'' Total Storage: 13972.59 GB Remaining Storage: 13094.16 GB Usage: 6.00% File Count: 1 files DRSU2: /dev/md0 Barcode: ''S15TCV13S0003'' Total Storage: 13972.59 GB Remaining Storage: 13272.59 GB Usage: 5.00% File Count: 0 files