mselog.txt showed temperature issues DP was shutdown, MCS was restarted, LE002 and LS001 were scheduled LE002 running until end of day File Information from DRSU LE002: 056026_000000205 056026 077468182 056027 005478182 DRX_FILT_7 001137380397216 001149979066368 YES 120410 01:37:37 056026_000000206 056026 077468182 056027 005478182 DRX_FILT_7 001136985970944 001149979066368 YES 120410 01:38:51 056026_000000207 056026 077468182 056027 005478182 DRX_FILT_7 001136782658688 001149979066368 YES 120410 01:39:43 056026_000000208 056026 077468182 056027 005478182 DRX_FILT_7 001137596791104 001149979066368 YES 120410 01:39:58 LS001: 120410 01:49:08 56027 6548396 T 3969 3 DR1 REC 056027_000003969| 120410 01:49:08 56027 6548396 T 3970 3 DR2 REC 056027_000003970| 03:51: 8.3 deg celsius outside temperature Observation finished: 056027_000003969 056027 006548182 056027 020958182 DRX_FILT_7 001138569550176 001149979066368 YES 120410 07:02:35 056027_000003970 056027 006548182 056027 020958182 DRX_FILT_7 001138569075456 001149979066368 YES 120410 07:03:20 06:59 starting TBN ./startTBN_split74.sh 11:04 Issued DP INI - TBN stop ./mesix DP_ INI ./mesix DP_ RPT SUMMARY LS001: 11:20 Observation start 120410 11:20:04 56027 40804537 T 7008 3 DR3 REC 056027_000007008| 120410 11:20:04 56027 40804537 T 7009 3 DR4 REC 056027_000007009| 15:22 Observation ended 056027_000007008 056027 040808182 056027 055218182 DRX_FILT_7 001138570132224 001149979066368 YES 120410 15:23:22 056027_000007009 056027 040808182 056027 055218182 DRX_FILT_7 001138570049664 001149979066368 YES 120410 15:23:25 15:26 starting TBN startTBN_split74.sh 20:03 Issued DP_ INI 20:05 ./startTBN_split74.sh 21:44 Current DRSU selection: DR1: VT DR2: VT DR3: UNM DR4: UNM DR5: UNM 23:00 Issued ASP INI 33, didn't seem to be initialized (brownout?) ./mesix ASP INI 33 ./setARX_split.sh 23:21 Powered FEEs ./mesix ASP FPW 000111 ./mesix ASP FPW 000211 ./mesix ASP FPW 259100 ./mesix ASP FPW 259200 Scheduled LS001 and LH001 for the night 56028 6307273 1 14410000 0 LS001 53 56028 6307273 2 14410000 0 LS001 54 56028 22879546 1 10810000 0 LH002 101 56028 22879546 2 10810000 0 LH002 102 56028 22879546 4 10810000 0 LH002 103 56028 40567273 3 14410000 0 LS001 55 56028 40567273 4 14410000 0 LS001 56 00:22 Added LC002 to queue to observe VirA and TauA until 4 hours after sunset 56028 6307273 1 14410000 0 LS001 53 56028 6307273 2 14410000 0 LS001 54 56028 22879546 1 10810000 0 LH002 101 56028 22879546 2 10810000 0 LH002 102 56028 22879546 4 10810000 0 LH002 103 56028 40567273 3 14410000 0 LS001 55 56028 40567273 4 14410000 0 LS001 56 56028 5395000 3 14410000 0 LC002 1 56028 5395000 4 14410000 0 LC002 2 01:15 Issued DP_ INI 01:30 LC002 starting 120411 01:29:55 56028 5395289 T 11421 3 DR3 REC 056028_000011421| 120411 01:29:55 56028 5395289 T 11422 3 DR4 REC 056028_000011422| two timeout messages with reference ids 11428, 11424 DRX seemed to confirm, no action taken 01:45 LS001 starting 120411 01:45:07 56028 6307529 T 11512 3 DR2 REC 056028_000011512| 120411 01:45:07 56028 6307999 T 11511 3 DR1 REC 056028_000011511| 02:23 Monitoring temperatures, the highest DP FPGA temperatures remain around 80C, but are not increasing. 04:21 DP: Warning condition cleared 05:31 Finished LC002 056028_000011421 056028 005400000 056028 019810000 DRX_FILT_7 001138568381952 001149979066368 YES 120411 05:38:39 056028_000011422 056028 005400000 056028 019810000 DRX_FILT_7 001138567230240 001149979066368 YES 120411 05:38:47 Note for LC002: Forgot pointing correction, will repeat observation in 24 hours. 05:45 Finished LS001 056028_000011511 056028 006312273 056028 020722273 DRX_FILT_7 001138570148736 001149979066368 YES 120411 05:47:12 056028_000011512 056028 006312273 056028 020722273 DRX_FILT_7 001138570103328 001149979066368 YES 120411 05:47:10 05:46 Issued DP_ INI 05:50 Copy logs to 120411 05:51 Change DRSU disk setting DR1: LWA (./mesix DR1 SEL 1) DR2: LWA (./mesix DR2 SEL 2) DR3: LWA DR4: LWA DR5: LWA 05:53 Change ASP filter mode to full ./setARX_full.sh 120411 05:55:08 56028 21308382 T 14005 1 ASP FIL 00001| 120411 05:55:08 56028 21308382 T 14005 2 ASP FIL 00001| 120411 05:55:09 56028 21309052 T 14005 3 ASP FIL | 120411 05:55:09 56028 21309387 T 14006 1 ASP AT1 00013| 120411 05:55:09 56028 21309387 T 14006 2 ASP AT1 00013| 120411 05:55:10 56028 21310362 T 14006 3 ASP AT1 | 120411 05:55:10 56028 21310391 T 14007 1 ASP AT2 00013| 120411 05:55:10 56028 21310391 T 14007 2 ASP AT2 00013| 120411 05:55:10 56028 21310690 T 14007 3 ASP AT2 | ATS = 15 06:21 Starting LH002 120411 06:21:21 56028 22881118 T 14095 3 DR4 REC 056028_000014095| 120411 06:21:21 56028 22881118 T 14094 3 DR2 REC 056028_000014094| 120411 06:21:21 56028 22881118 T 14093 3 DR1 REC 056028_000014093| 09:21 LH002 stopped 056028_000014093 056028 022884546 056028 033694546 DRX_FILT_7 000853270099584 000862683922432 YES 120411 09:45:38 056028_000014094 056028 022884546 056028 033694546 DRX_FILT_7 000853601776128 000862683922432 YES 120411 09:46:24 056028_000014095 056028 022884546 056028 033694546 DRX_FILT_7 000853471624416 000862683922432 YES 120411 09:47:04 09:30 DP_ INI (at) 09:31 setARX_split.sh (at) 120411 09:30:00 56028 34200577 T 16141 1 DP_ INI | 120411 09:30:00 56028 34200577 T 16141 2 DP_ INI | 120411 09:30:00 56028 34200690 T 16141 3 DP_ INI | 120411 09:31:00 56028 34260529 T 16142 1 ASP FIL 00000| 120411 09:31:00 56028 34260529 T 16142 2 ASP FIL 00000| 120411 09:31:01 56028 34261334 T 16142 3 ASP FIL | 120411 09:31:01 56028 34261532 T 16143 1 ASP ATS 00015| 120411 09:31:01 56028 34261532 T 16143 2 ASP ATS 00015| 120411 09:31:01 56028 34261666 T 16143 3 ASP ATS | 120411 09:31:02 56028 34262535 T 16144 1 ASP AT1 00008| 120411 09:31:02 56028 34262535 T 16144 2 ASP AT1 00008| 120411 09:31:02 56028 34262630 T 16144 3 ASP AT1 | 120411 09:31:03 56028 34263539 T 16145 1 ASP AT2 00006| 120411 09:31:03 56028 34263539 T 16145 2 ASP AT2 00006| 120411 09:31:04 56028 34264440 T 16145 3 ASP AT2 | 09:35 startTBN_split74.sh (at) 120411 09:31:42 56028 34302514 T 16152 8 SHL SHT Timed out at subsystem| 120411 09:35:00 56028 34500590 T 16164 3 DP_ RPT NORMAL 09:40 Change DRSU setting DR1: VT (./mesix DR1 SEL 2) DR2: VT (./mesix DR2 SEL 1) DR3: VT (./mesix DR3 SEL 1) DR4: VT (./mesix DR4 SEL 2) DR5: UNM 11:00 DP_ INI (at)/PASI stopped 15:25 DP_ INI (at) 11:16 LS001 Observations start 120411 11:16:12 56028 40572350 T 16539 3 DR3 REC 056028_000016539| 120411 11:16:12 56028 40572350 T 16540 3 DR4 REC 056028_000016540| 120411 11:16:42 56028 40602385 T 16553 8 SHL SHT Timed out at subsystem| Final State: 3 No action taken 15:16 LS001 observation finished copied logs 056028_000016539 056028 040572273 056028 054982273 DRX_FILT_7 001137770369376 001149979066368 YES 120411 19:32:42 056028_000016540 056028 040572273 056028 054982273 DRX_FILT_7 001138165167168 001149979066368 YES 120411 19:32:45 15:27 DP_ INI 15:43 ./startTBN_split74.sh 19:55 DP_ INI 19:00-21:54 Maintenance (Chris & Jayce) 22:00 Restart MCS ./meeix SHT ./msei MCS SHT saved logfiles ./start_station.sh ./me_init 0 Select DRSUs: DR1: ./mesix DR1 SEL 2 (VT) DR2: ./mesix DR2 SEL 1 (VT) DR3: ./mesix DR3 SEL 2 (UNM) DR4: ./mesix DR4 SEL 1 (UNM) DR5: ./mesix DR5 SEL 1 (UNM) Filled the schedule for the evening: S PROJ SESS B MJD MPM DUR NOBS C 0 3 LC002 3 3 56029 5395000 14410000 4 0 LC002_0003.ses 1 3 LC002 4 4 56029 5395000 14410000 4 0 LC002_0004.ses 2 3 LS001 57 1 56029 6071364 14410000 1 0 LS001_0057.ses 3 3 LS001 58 2 56029 6071364 14410000 1 0 LS001_0058.ses 4 3 LH002 104 1 56029 22643637 10810000 1 0 LH002_0104.ses 5 3 LH002 105 2 56029 22643637 10810000 1 0 LH002_0105.ses 6 3 LH002 106 4 56029 22643637 10810000 1 0 LH002_0106.ses 22:14 Start PASI 01:15 DP_ INI (at) 01:30 LC002 begins 120412 01:29:55 56029 5395264 T 789 3 DR3 REC 056029_000000789| 120412 01:29:55 56029 5395265 T 790 3 DR4 REC 056029_000000790| 01:41 LS001 begins 120412 01:41:11 56029 6071622 T 873 3 DR2 REC 056029_000000873| 120412 01:41:12 56029 6072049 T 872 3 DR1 REC 056029_000000872| 05:30 LC002 ends 056029_000000789 056029 005400000 056029 019810000 DRX_FILT_7 001138568262240 001149979066368 YES 120412 05:31:27 056029_000000790 056029 005400000 056029 019810000 DRX_FILT_7 001138566454176 001149979066368 YES 120412 05:31:24 05:41 LS001 ends 056029_000000872 056029 006076364 056029 020486364 DRX_FILT_7 001138569950592 001149979066368 YES 120412 05:44:45 056029_000000873 056029 006076364 056029 020486364 DRX_FILT_7 001138572394368 001149979066368 YES 120412 05:44:42 06:00 DP_ INI switch to LWA DRSUs on DR1,DR2 save logfiles change filter settings to full at (615) 120412_0600.sh 06:17 LH002 begins 120412 06:17:25 56029 22645291 T 3456 3 DR4 REC 056029_000003456| 120412 06:17:25 56029 22645291 T 3454 3 DR1 REC 056029_000003454| 120412 06:17:25 56029 22645291 T 3455 3 DR2 REC 056029_000003455| 09:17 LH002 ends 056029_000003454 056029 022648637 056029 033458637 DRX_FILT_7 000854125433568 000862683922432 YES 120412 18:54:33 056029_000003455 056029 022648637 056029 033458637 DRX_FILT_7 000854125342752 000862683922432 YES 120412 18:56:00 056029_000003456 056029 022648637 056029 033458637 DRX_FILT_7 000854124843264 000862683922432 YES 120412 18:56:32 09:20 Switch DR1-DR4 to VT DRSUs DP_ INI Change filter settings save copy of logs at (616) 120412_0920.sh 09:25 start PASI (at 617 ./startTBN_split74.sh) 09:27 beamformer calibration failed, reissued DP_ INI 09:36 startTBN_split74.sh 12:30 DP _INI (at 618 INIdp.sh) 12:44 LE002 begins at 620 06:44 04/12/12 -f 120412_CygA_1246.sh DR4 SPC 056029_000006198| DR3 SPC 056029_000006194| DR1 SPC 056029_000006186| DR2 SPC 056029_000006190| 14:46 LE002 ends 14:50 Switch filter setting to split, select DR1-DR4 UNM DRSUs at 619 120412_1450_SELDRSUs.sh 14:55 COMM begins at 621 08:54 04/12/12 -f 120412_1455_b3_out.sh at 622 08:55 04/12/12 -f 120412_1455_b1_CasA.sh at 623 08:56 04/12/12 -f 120412_1455_b2_CygA.sh Scripts failed needed fixing Beam 2, Beam 3 seem to work fine Beam 1 running with error: 120412 15:07:08 56029 54428247 T 6747 9 DP_ BAM Can't open '../sch/gfiles/111226_XY.gf'| 120412 14:55:00 56029 53700856 T 6657 3 DR1 REC 056029_000006657| ENTRY-3 120412 14:56:00 56029 53760876 T 6678 3 DR2 REC 056029_000006678| ENTRY-3 120412 15:06:26 56029 54386133 T 6744 3 DR3 REC 056029_000006744| ENTRY-3 120412 15:07:08 56029 54428522 T 6748 3 DR1 REC 056029_000006748| ENTRY-4 120412 15:08:34 56029 54514549 T 6752 3 DR2 REC 056029_000006752| ENTRY-4 15:10 Recording started 16:18 Reissuing the BAM didn't help, diagnosing the problem gain file missing for beam 1, data can be deleted 17:15 COMM ends 17:25 DP_ INI at 626 11:25 04/12/12 -f INIdp.sh 17:35 Start PASI at 627 11:35 04/12/12 -f startTBN_split74.sh 17:40 Saved Logs 19:40 Cleared DR queue clearing SCHEDULE entries Deleted from DR1: 056029_000006657, 056029_000006748 Kept on DR2 the obsolete file: 056029_000006678 19:55 DP_ INI -> hand over to Steve for maintenance 21:20 Station returned -> start MCS Started scheduler Started executive 21:22 ./startTBN_split74.sh 21:23 Selected DRSUs: DR1: LWA DR2: LWA DR3: VT DR4: VT DR5: LWA 120412 22:25:51 56029 80751290 T 232 3 DP_ RPT T_NOM1! 0x07! Beamformer calibration lost - MGT underflow(s) detected| 22:25 DP_INI 22:37 ./startTBN_split74.sh 23:49 Filled observation queue until maintenance S PROJ SESS B MJD MPM DUR NOBS C 0 3 LC002 5 1 56030 5395000 14410000 4 0 LC002_0005.ses 1 3 LC002 6 2 56030 5395000 14410000 4 0 LC002_0006.ses 2 3 LS001 59 3 56030 5835455 14410000 1 0 LS001_0059.ses 3 3 LS001 60 4 56030 5835455 14410000 1 0 LS001_0060.ses 4 3 LH002 107 1 56030 22407728 10810000 1 0 LH002_0107.ses 5 3 LH002 108 2 56030 22407728 10810000 1 0 LH002_0108.ses 6 3 LH002 109 3 56030 22407728 10810000 1 0 LH002_0109.ses 7 3 LS001 61 3 56030 40095455 14410000 1 0 LS001_0061.ses 8 3 LS001 62 4 56030 40095455 14410000 1 0 LS001_0062.ses 01:15 DP_ INI (628) at 19:15 04/12/12 -f INIdp.sh 01:30 LC002 begins 120413 01:29:55 56030 5395297 T 865 3 DR2 REC 056030_000000865| 120413 01:29:55 56030 5395297 T 864 3 DR1 REC 056030_000000864| 01:37 LS001 begins 120413 01:37:16 56030 5836562 T 921 3 DR3 REC 056030_000000921| 120413 01:37:16 56030 5836562 T 922 3 DR4 REC 056030_000000922| 120413 01:46:06 56030 6366218 T 994 8 SHL SHT Timed out at subsystem| wasn't DP_ 05:30 LC002 ends 056030_000000864 056030 005400000 056030 019810000 DRX_FILT_7 001138568423232 001149979066368 YES 120413 05:30:44 056030_000000865 056030 005400000 056030 019810000 DRX_FILT_7 001138567952640 001149979066368 YES 120413 05:31:00 05:37 LS001 ends 056030_000000921 056030 005840455 056030 020250455 DRX_FILT_7 001138330117920 001149979066368 YES 120413 05:38:04 056030_000000922 056030 005840455 056030 020250455 DRX_FILT_7 001138346485440 001149979066368 YES 120413 05:38:10 05:50 change filter settings, copy logs, select LWA disks, DP_ INI (629) at 23:50 04/12/12 -f 120413_0550.sh 05:55 DP_ INI (630) at 23:55 04/12/12 -f INIdp.sh 06:13 LH002 begins 120413 06:13:29 56030 22409617 T 3524 3 DR1 REC 056030_000003524| 120413 06:13:29 56030 22409617 T 3526 3 DR3 REC 056030_000003526| 120413 06:13:29 56030 22409617 T 3525 3 DR2 REC 056030_000003525| 120413 06:13:45 56030 22425000 T 3529 8 DP_ SHT Timed out at subsystem| 120413 06:13:45 56030 22425000 N Previous message: eMIBerror=64 120413 06:13:45 56030 22425000 T 3527 8 DP_ SHT Timed out at subsystem| 120413 06:13:45 56030 22425000 N Previous message: eMIBerror=64 NOT DP_ DRX no action taken 09:13 LH002 ends 09:20 change filter settings, copy logs, change disk settings, DP_ INI (631) at 03:20 04/13/12 -f 120413_0920.sh (632) at 03:21 04/13/12 -f INIdp.sh 09:30 start PASI (633) at 03:30 04/13/12 -f ./startTBN_split74.sh 10:55 DP_ INI (634) at 04:55 04/13/12 -f INIdp.sh 11:08 LS001 begins 120413 11:08:15 56030 40095722 T 5946 3 DR3 REC 056030_000005946| 120413 11:08:15 56030 40095722 T 5947 3 DR4 REC 056030_000005947| 15:08 LS001 ends DRX_FILT_7 1138569752448 Apr 13 15:08 056030_000005946 DRX_FILT_7 1138570115712 Apr 13 15:08 056030_000005947 15:15 DP_ INI (635) at 09:15 04/13/12 -f INIdp.sh 15:25 start PASI (636) at 09:25 04/13/12 -f ./startTBN_split74.sh MAINTENANCE Ended 17:30 01:15 DP_ INI at 19:15 04/13/12 -f INIdp.sh 01:30 LC002 begins 120414 01:29:55 56031 5395436 T 2339 3 DR2 REC 056031_000002339| 120414 01:29:55 56031 5395437 T 2338 3 DR1 REC 056031_000002338| 05:30 LC002 ends 056031_000002338 056031 005400000 056031 019810000 DRX_FILT_7 001138567952640 001149979066368 YES 120414 23:47:02 056031_000002339 056031 005400000 056031 019810000 DRX_FILT_7 001138566293184 001149979066368 YES 120414 23:47:11 05:35 DP_ INI 05:42 start PASI 14:00 DP_ INI (643) at 08:00 04/14/12 -f ./INIdp.sh 14:10 script1 (645) at 08:10 04/14/12 -f ./120414_1415_b1_CasA.sh 14:11 script2 (646) at 08:11 04/14/12 -f ./120414_1415_b2_CygA.sh 14:12 script3 (647) at 08:12 04/14/12 -f ./120414_1415_b3_out.sh 14:15 COMM starts 120414 14:10:01 56031 51001372 T 5520 3 DR1 REC 056031_000005520| 120414 14:11:01 56031 51061400 T 5524 3 DR2 REC 056031_000005524| 120414 14:12:00 56031 51120439 T 5545 3 DR3 REC 056031_000005545| 14:30 Begin observations 16:30 End observations 056031_000005520 056031 051300000 056031 058500000 DRX_FILT_7 000568890016512 000574590812160 YES 120414 23:54:02 056031_000005524 056031 051300000 056031 058500000 DRX_FILT_7 000568889995872 000574590812160 YES 120414 23:54:08 056031_000005545 056031 051300000 056031 058500000 DRX_FILT_7 000568890066048 000574590812160 YES 120414 23:54:20 17:00 DP_INI (648) at 11:00 04/14/12 -f ./INIdp.sh 17:10 Start PASI (649) at 11:10 04/14/12 -f ./startTBN_split74.sh Lots of RFI between UT 18--24h Lots of RFI between UT 18--24hr 00:28 Submitted LC002 to queue S PROJ SESS B MJD MPM DUR NOBS C 0 3 LC002 9 3 56032 5395000 14410000 4 0 LC002_0009.ses 1 3 LC002 10 4 56032 5395000 14410000 4 0 LC002_0010.ses 00:30 DP_ INI 01:15 DP_ INI (650) at 19:15 04/14/12 -f ./INIdp.sh 01:30 LC002 begins At least the beginning, very likely contaminated by a lot of RFI. 120415 01:30:00 56032 5400245 T 7890 3 DR3 REC 056032_000007890| 120415 01:30:00 56032 5400245 T 7891 3 DR4 REC 056032_000007891| 120415 01:30:15 56032 5415268 T 7893 8 DP_ SHT Timed out at subsystem| 120415 01:30:15 56032 5415268 N Previous message: eMIBerror=64 120415 01:30:15 56032 5415268 T 7894 8 DP_ SHT Timed out at subsystem| 120415 01:30:15 56032 5415268 N Previous message: eMIBerror=64 120415 01:30:20 56032 5420276 T 7893 3 DP_ DRX | 120415 01:30:20 56032 5420276 N Previous message: eMIBerror=12 120415 01:30:25 56032 5425280 T 7894 3 DP_ DRX | 120415 01:30:25 56032 5425280 N Previous message: eMIBerror=4 120415 01:30:30 56032 5430000 T 7898 5 DP_ RPT Timed out at ms_mcic| State 3, no action taken 05:30 LC002 ends 056032_000007890 056032 005400000 056032 019810000 DRX_FILT_7 001138568538816 001149979066368 YES 120415 05:59:28 056032_000007891 056032 005400000 056032 019810000 DRX_FILT_7 001138568604864 001149979066368 YES 120415 05:59:47 05:35 DP_ INI (651) at 23:35 04/14/12 -f ./INIdp.sh 04:45 Start PASI (652) at 23:45 04/14/12 -f ./startTBN_split74.sh 12:15 DP_ INI (653) at 06:15 04/15/12 -f ./INIdp.sh Still weaker RFI to the North-East. 12:32 COMM begins (654) at 06:32 04/15/12 -f 120415_CygA_1234.sh DR1 SPC 056032_000010766| DR2 SPC 056032_000010770| DR3 SPC 056032_000010774| DR4 SPC 056032_000010778| 14:34 COMM ends (657) at 06:36 04/15/12 -f setARX_split.sh screwed up the time, reset filter settings before it had to. 14:40 DP_ INI (655) at 08:40 04/15/12 -f ./INIdp.sh 14:50 Start PASI (656) at 08:50 04/15/12 -f ./startTBN_split74.sh Runnig PASI until end of the day Since powerup some rogue process was running on DP causing intermittent problems.