ALMA Pipeline Known Issues

Browser security and viewing file://weblog

More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you:

Known Issues for PL2021.2.0

Release date Oct 2021

CASA

1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset.

Interferometric Pipeline

1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability.

2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines.

3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator).

4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data).

5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal.

6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).

7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.

8. full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.

Single Dish Pipeline

0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).

Known Issues for PL2020.1.0

Release date Feb 2021

CASA

CASA 6.1.X known issues

Interferometric Pipeline

0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.

1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.

2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.

3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.

4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.

5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.

6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.

7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run.

8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.

9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data.

10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging, hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)

11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect.

12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.

13. full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.

14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).

Single Dish Pipeline

1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.

2. Strong emission is flagged in hsd_blflag. The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually.

3. Wrong position of the open circle indicating peak SN position in Peak SN map. When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.

4. hsdn_restoredata crashes (Nobeyama Pipeline)

5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)