https://casaguides.nrao.edu/api.php?action=feedcontributions&user=Thunter&feedformat=atomCASA Guides - User contributions [en]2024-03-29T11:59:15ZUser contributionsMediaWiki 1.38.6https://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=36739ALMA Pipeline Known Issues2024-03-13T19:19:50Z<p>Thunter: /* Browser security and viewing file://weblog */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
13. In hifa_bandpass, the plots of phase vs. frequency can take a long time to display in the weblog due to a change in behavior of a mathematical function due to an upgrade in the scipy version packaged with the pipeline ([https://open-jira.nrao.edu/browse/PIPE-2035 PIPE-2035]).<br />
<br />
14. In full-polarization datasets, the hif_makeimages page for the polcal has images of polarized intensity; the ">_" link which usually produces a popup with the tclean command, in this case just dims the screen without revealing a tclean command, because the image was not made with tclean. Clicking the image background will refresh the screen. <br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors (see [https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp | Firefox security update details]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of <b>firefox</b> (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting <b>"security.fileuri.strict_origin_policy"</b> to False). For google chrome, it can be started with the following command line arguments: <b>chromium-browser --disable-web-security --user-data-dir="[some directory here]"</b><br />
<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
16. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
18. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=36738ALMA Pipeline Known Issues2024-03-13T19:17:12Z<p>Thunter: /* Browser security and viewing file://weblog */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
13. In hifa_bandpass, the plots of phase vs. frequency can take a long time to display in the weblog due to a change in behavior of a mathematical function due to an upgrade in the scipy version packaged with the pipeline ([https://open-jira.nrao.edu/browse/PIPE-2035 PIPE-2035]).<br />
<br />
14. In full-polarization datasets, the hif_makeimages page for the polcal has images of polarized intensity; the ">_" link which usually produces a popup with the tclean command, in this case just dims the screen without revealing a tclean command, because the image was not made with tclean. Clicking the image background will refresh the screen. <br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors (see [https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp|Firefox security update details]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of <b>firefox</b> (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting <b>"security.fileuri.strict_origin_policy"</b> to False). For google chrome, it can be started with the following command line arguments: <b>chromium-browser --disable-web-security --user-data-dir="[some directory here]"</b><br />
<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
16. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
18. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=36136ALMA Pipeline Known Issues2023-12-13T17:33:50Z<p>Thunter: /* Browser security and viewing file://weblog */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
13. In hifa_bandpass, the plots of phase vs. frequency can take a long time to display in the weblog due to a change in behavior of a mathematical function due to an upgrade in the scipy version packaged with the pipeline ([https://open-jira.nrao.edu/browse/PIPE-2035 PIPE-2035]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of <b>firefox</b> (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting <b>"security.fileuri.strict_origin_policy"</b> to False). For google chrome, it can be started with the following command line arguments: <b>chromium-browser --disable-web-security --user-data-dir="[some directory here]"</b><br />
<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
16. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
18. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=36135ALMA Pipeline Known Issues2023-12-13T17:33:35Z<p>Thunter: /* Browser security and viewing file://weblog */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
13. In hifa_bandpass, the plots of phase vs. frequency can take a long time to display in the weblog due to a change in behavior of a mathematical function due to an upgrade in the scipy version packaged with the pipeline ([https://open-jira.nrao.edu/browse/PIPE-2035 PIPE-2035]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of <b>firefox<b> (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting <b>"security.fileuri.strict_origin_policy"<b> to False). For google chrome, it can be started with the following command line arguments: <b>chromium-browser --disable-web-security --user-data-dir="[some directory here]"</b><br />
<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
16. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
18. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=36134ALMA Pipeline Known Issues2023-12-13T17:33:06Z<p>Thunter: /* Browser security and viewing file://weblog */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
13. In hifa_bandpass, the plots of phase vs. frequency can take a long time to display in the weblog due to a change in behavior of a mathematical function due to an upgrade in the scipy version packaged with the pipeline ([https://open-jira.nrao.edu/browse/PIPE-2035 PIPE-2035]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False). For google chrome, it can be started with the following command line arguments: <b>chromium-browser --disable-web-security --user-data-dir="[some directory here]"</b><br />
<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
16. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
18. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=36133ALMA Pipeline Known Issues2023-12-13T17:32:38Z<p>Thunter: /* Browser security and viewing file://weblog */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
13. In hifa_bandpass, the plots of phase vs. frequency can take a long time to display in the weblog due to a change in behavior of a mathematical function due to an upgrade in the scipy version packaged with the pipeline ([https://open-jira.nrao.edu/browse/PIPE-2035 PIPE-2035]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False). For google chrome, it can be started with the following command line arguments: chromium-browser --disable-web-security --user-data-dir="[some directory here]"<br />
<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
16. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
18. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=36022ALMA Pipeline Known Issues2023-11-13T04:04:37Z<p>Thunter: /* IF */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
13. In hifa_bandpass, the plots of phase vs. frequency can take a long time to display in the weblog due to a change in behavior of a mathematical function due to an upgrade in the scipy version packaged with the pipeline ([https://open-jira.nrao.edu/browse/PIPE-2035 PIPE-2035]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
16. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
18. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35976ALMA Pipeline Known Issues2023-10-31T17:25:22Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
16. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
18. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35975ALMA Pipeline Known Issues2023-10-31T17:25:06Z<p>Thunter: /* PL2021.2.0 */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
16. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35974ALMA Pipeline Known Issues2023-10-31T17:24:28Z<p>Thunter: /* PL2023 */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
--- Items below this point were discovered after the PL2023 google slides were released and frozen: ---<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35973ALMA Pipeline Known Issues2023-10-31T17:24:08Z<p>Thunter: /* PL2023 */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
*Items below this point were discovered after the PL2023 google slides were released and frozen:*<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35972ALMA Pipeline Known Issues2023-10-31T17:23:08Z<p>Thunter: /* PL2023 */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat ([https://open-jira.nrao.edu/browse/PIPE-1898 PIPE-1898])<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
3. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
4. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
5. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
6. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1664 PIPE-1664]). <br />
<br />
7. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on only 1 of the transfer targets (typically either PHASE or CHECK source) and it still exhibits the best score in hif_refant ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
8. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
9. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
10. In hifa_spwphaseup, (i) if the refant drops out for the majority of solutions, then the Median Phase RMS assessment cannot be made; (ii) if there is low SNR in the bandpass solutions and the refant jumps in phase, then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. <br />
<br />
11. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
12. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35971ALMA Pipeline Known Issues2023-10-31T17:09:56Z<p>Thunter: /* PL2022.2 */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
19. In hif_findcont, spws with a single range of continuum channels identified might be declared AllCont even if they contain a significantly lesser amount of continuum than the nominal threshold of 91% ([https://open-jira.nrao.edu/browse/PIPE-2034 PIPE-2034]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35315ALMA Pipeline Known Issues2023-09-08T12:26:32Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35314ALMA Pipeline Known Issues2023-09-08T12:26:11Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35313ALMA Pipeline Known Issues2023-09-08T12:25:40Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]). The work-around is to set edgespw=False in hifa_flagdata.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35311ALMA Pipeline Known Issues2023-09-06T17:44:08Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
17. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35310ALMA Pipeline Known Issues2023-09-06T17:43:22Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909).<br />
<br />
15. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35309ALMA Pipeline Known Issues2023-09-06T17:43:02Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
18. ACA projects with multiple single-spw FDM tunings might experience odd flagging of non-edge channels ([https://open-jira.nrao.edu/browse/PIPE-1991 PIPE-1991]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909). <br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35307ALMA Pipeline Known Issues2023-08-15T23:22:14Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
17. For heterogeneous array mosaics with 3 or more PM antennas included, if any field has only 1 baseline of data left unflagged after online flags and any prior stage flagging, then hifa_targetflag will crash ([https://open-jira.nrao.edu/browse/PIPE-1879 PIPE-1879]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909). <br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35306ALMA Pipeline Known Issues2023-08-15T02:02:06Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623]).<br />
* Inaccurate warning and QA score in hifa_spwphaseup (PIPE-1625).<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909). <br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35305ALMA Pipeline Known Issues2023-08-15T01:54:26Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
<br />
=== CASA ===<br />
1. ephemeris imaging in MPIcasa remains unreliable. ''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is still not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
<br />
=== Both Pipelines ===<br />
1. PL2023 uses python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
=== IF ===<br />
1. hif_uvcontsub will not skip spws that have "NONE" in cont.dat. This situation doesn't arise in operations, but could arise if someone manually edits cont.dat (PIPE-1898)<br />
<br />
2. hif_selfcal will fail for multi-EB single field projects with virtual spws - this is quite rare outside of polarization, and self-calibration of full-polarization projects is not fully validated in any case.<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623])<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0 ([https://open-jira.nrao.edu/browse/PIPE-1633 PIPE-1633]). (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909). <br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35297ALMA Pipeline Known Issues2023-07-30T20:37:30Z<p>Thunter: /* PL2021.2.0 */</p>
<hr />
<div>== PL2023 ==<br />
1. PL2023 will use python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623])<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
14. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning (PIPE-1909). <br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35296ALMA Pipeline Known Issues2023-07-30T20:37:01Z<p>Thunter: /* PL2022.2 */</p>
<hr />
<div>== PL2023 ==<br />
1. PL2023 will use python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623])<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35295ALMA Pipeline Known Issues2023-07-30T20:36:28Z<p>Thunter: </p>
<hr />
<div>== PL2023 ==<br />
1. PL2023 will use python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623])<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
15. In the polcal recipe recipe, hifa_gfluxscale can fail silently (pausing pipeline permanently) if the refant has 100% flagging on either PHASE or CHECK source. (PIPE-1805). The workaround is to set geometry=False in hif_refant.<br />
<br />
16. On the Spectral Setup weblog page, multi-target datasets with multiple tunings not observed in all targets will not show the Transition names for spws associated with tunings beyond the first tuning ([https://open-jira.nrao.edu/browse/PIPE-1909 PIPE-1909]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35001ALMA Pipeline Known Issues2023-04-24T16:33:54Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
1. PL2023 will use python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623])<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial. <br />
<br />
12. Phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected ([https://open-jira.nrao.edu/browse/PIPE-1661 PIPE-1661]).<br />
<br />
13. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
14. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=35000ALMA Pipeline Known Issues2023-04-24T16:30:38Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== PL2023 ==<br />
1. PL2023 will use python3.8. Some user machines may find dependency issues if they only have python3.6 available. <br />
<br />
<br />
----<br />
----<br />
<br />
== PL2022.2 ==<br />
2022.2.0.64 used Oct 2022 - April 2023<br />
<br />
2022.2.0.68 used April 2023 - Oct 2024 fixes some of the issues below, as noted.<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|50px]]<br />
''PLWG recommends that due to this, all multi-EB ephemeris data be run in serial mode.''<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Both Pipelines ===<br />
<br />
PL2022 doesn't import on MacOS - PL was never fully supported on MacOS, and unfortunately this one doesn't work at all. '''''Fixed in 2022.2.0.68'''''<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575] -> ''fixed PL2023'').<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623])<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it '''rarely''' means that continuum subtraction needs to be redone, but merely that in some circumstances the continuum image can be improved to reduce line contamination (see DRM training material).<br />
<br />
11. In the polcal recipe, if the selected session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR. In a related issue, if the session reference antenna is fully flagged due to shadowing on only 1 of the transfer targets (and still wins the best score in hif_refant), then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1805 PIPE-1805]).<br />
<br />
12. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708]).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
1. The "npiece" parameter determining number of cubic splines to fit can sometimes be miscalculated when the baseline is complex, leading to non-optimal baseline fits. ([https://open-jira.nrao.edu/browse/PIPE-1773 PIPE-1773], '''''Fixed in 2022.2.0.68''''')<br />
<br />
2. Weights calculated for spectra completely flagged in only one polarization can be inappropriately high, leading to bright and dark artifacts in the images. ([https://open-jira.nrao.edu/browse/PIPE-1771 PIPE-1771], '''''Fixed in 2022.2.0.68''''' ''affected data will be reprocessed'')<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
----<br />
<br />
== PL2021.2.0 ==<br />
Oct 2021 - Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities&diff=34996Analysis Utilities2023-04-12T11:22:40Z<p>Thunter: /* Getting Started */</p>
<hr />
<div>[[Category:CASA]] [[Category:ALMA]] [[Category:EVLA]]<br />
Analysis Utilities (or analysisUtils for short) is a small set of Python source code files that provide a number of analysis and plotting utilities. The primary file is analysisUtils.py (108000 lines as of March 2021). The utilities were developed (and continue to be developed) for ALMA commissioning and data reduction and are, in many cases, also useful for EVLA data reduction. This CASA Guide documents some of the most useful functions contained in the analysisUtils Python module from a user's perspective, but most of the functions have fairly complete inline help. <br />
<br />
=Getting Started=<br />
<br />
'''If you are working on a machine at NRAO Charlottesville, then you can skip Step 1, because the latest modules are already available at: /users/thunter/AIV/science/analysis_scripts. ''' <br />
<br />
'''Step 1:''' Download Analysis Utilities <br />
There are two options:<br />
<br />
a) [ftp://ftp.cv.nrao.edu/pub/casaguides/analysis_scripts.tar NRAO ftp server]. (If your browser does not want to open an ftp link, then try right click to save the link and select the application to use to be wget.) <br />
<br />
b) [https://zenodo.org/record/7502160 zenodo] (this provides a citeable DOI: 10.5281/zenodo.7502160, on ADS see [https://ui.adsabs.harvard.edu/abs/2023zndo...7502160H/abstract 2023zndo...7502160H])<br />
<br />
Once downloaded, then extract the tar ball. From a Unix command line this can be done with<br />
<br />
<source lang="bash"><br />
$ tar xvf analysis_scripts.tar<br />
</source><br />
<br />
Seven files will be extracted -- a README file, and six python scripts. The README file contains the time<br />
and date that the tar ball was generated, which can be useful for reporting bugs. The [ftp://ftp.cv.nrao.edu/pub/casaguides/history history] file available at the ftp site contains a list of major changes in each version. <br />
<br />
'''Step 2:''' Edit your existing casa initialization file (in CASA5: <tt>$HOME/.casa/init.py</tt>, or in CASA6: <tt>$HOME$/.casa/startup.py</tt>) or create a new file if it does not already exist, and add the following<br />
<br />
<source lang="Python"><br />
import sys<br />
sys.path.append("/PATH_TO_ANALYSIS_SCRIPTS/")<br />
import analysisUtils as au<br />
</source><br />
<br />
where <tt>/PATH_TO_ANALYSIS_SCRIPTS/</tt> is the path to the directory you just extracted from the tar ball. An example can be downloaded and edited here: [[File:Init.py]]. When this is done, start casapy and you will have access to all the functions contained in the analysisUtils module.<br />
<br />
=Support=<br />
<br />
Once installed, functions can be searched for by name via minimum match string with the au.help function (where comma denotes a logical AND): e.g. au.help('antenna') or au.help('spw,caltable'). As in python, inline help for an individual function is accessed by typing 'help au.planet'. For further help with analysisUtils installation or usage, you may submit a helpdesk ticket to the Data Reduction category at https://help.almascience.org/. Be advised that analysisUtils is a commissioning tool, so the level of support available is not as extensive as for CASA, but we do welcome feedback. If your question is with regard to [[plotbandpass]], then please provide the version number of this program (it is printed to the screen at execution time and on the plots). In all cases, please provide the release date of your analysisUtils, which can be viewed from inside casapy as follows:<br />
<source lang="Python"><br />
au.version()<br />
</source><br />
<br />
=Script generators=<br />
The documentation for the two script generators almaqa2csg.py and almaqa2isg.py, as well as how to provide feedback or get support for their usage, can be found at this link: https://confluence.alma.cl/display/EAPR/ALMA+Data+Analysis+Utilities<br />
<br />
=Some Examples of Functions=<br />
<b>Up-to-date, online help is available for all of these functions (and more) at the CASA prompt by simply using the command: help(au.functionName).</b> To search for functions, you can type au.help('phrase'), where phrase is case-insensitive and minimum-match. Multiple phrases can be combined with a comma, for example au.help('baseline,stats').<br />
; [[angularSeparationOfFields]]<br />
: reads an ms and computes the angular separation of all fields, or a subset<br />
; [[buildConfigurationFile]]<br />
: reads the list of antenna stations in an .ms and creates a .cfg file suitable for [http://casa.nrao.edu/docs/taskref/simobserve-task.html simobserve]<br />
; [[editIntents]]<br />
: change the intents for a specified field in an ms (based on John Lightfoot's pipeline script)<br />
; [[getBaselineLengths]]<br />
: returns a dictionary of the baseline lengths in your ms, by default sorted by length<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/GetBaselineStats getBaselineStats]<br />
: returns a bunch of statistics on the baseline lengths in your ms<br />
; [[gjincBeam]]<br />
: compute the effective restoring beam obtained from the casa command {{sdimaging}} when using the GJINC gridding kernel<br />
; [[lstrange]]<br />
: lists the range of LST, UT, MJD for the whole ms, and for scans with OBSERVE_TARGET intent (including the elevation range)<br />
; [[obslist]]<br />
: prints the antenna station coordinates in local offsets from the Center of Array, and computes longest/shortest baselines<br />
; [[planet]]<br />
: contacts the JPL Horizons telnet service and returns the J2000 position, velocity, angular diameter, range and range rate of a solar system object<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/PlanetFlux planetFlux]<br />
: calls Bryan Butler's solar_system_setjy module in CASA to compute flux density vs. time or flux density vs. frequency <br />
; [[planetPlots]]<br />
: generates a plots of uv amplitude vs. uv distance for a grid of ALMA configurations and observing frequencies<br />
; [[plotbandpass]] <br />
: a faster version of plotcal for bandpass tables, with useful overlay capabilities. This function has been incorporated into CASA as a task.<br />
; [[plotSunDuringTrack]]<br />
: generates a plot of the Sun's az vs. el during an ms or ASDM<br />
; [[plotconfig]] <br />
: plot any standard telescope configuration of observatories known to casa, and return an array of the sorted baseline lengths<br />
; [[plotmosaic]]<br />
: shows relative location of pointings in an .ms<br />
; [[plotPWV]]<br />
: reads the PWV from each antenna's WVR from the ASDM_CALWVR table in an ms, and creates a plot vs. time<br />
; [[plotweather]]<br />
: plot weather conditions vs. time for your ms<br />
; [[plotWVRSolutions]]<br />
: plot the WVR solutions in terms of baseline-based phase corrections<br />
; [[spectralindex]]<br />
: performs a least-squares fit to the multi-spw output from {{fluxscale}}<br />
; [[timeOnSource]]<br />
: compute the total time spent integrating on-source for each specified field<br />
<br />
<b>A list of additional 1200+ functions, and their brief descriptions, can be found at [https://safe.nrao.edu/wiki/bin/view/Main/CasaExtensions Todd's wikipage]</b></div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=34138ALMA Pipeline Known Issues2023-01-19T16:16:51Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Known Issues for PL2022.2.0 ==<br />
Release date Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|100px]]<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575]).<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623])<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment. (iv) The phase structure function plots are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it rarely means that continuum subtraction needs to be redone (see DRM training material)<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial mode.<br />
<br />
12. In the polcal recipe, if the session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR.<br />
<br />
13. hifa_imageprecheck will crash on mosaic observations of science targets that have integer names ([https://open-jira.nrao.edu/browse/PIPE-1708 PIPE-1708])<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities&diff=34006Analysis Utilities2023-01-14T19:24:13Z<p>Thunter: /* Getting Started */</p>
<hr />
<div>[[Category:CASA]] [[Category:ALMA]] [[Category:EVLA]]<br />
Analysis Utilities (or analysisUtils for short) is a small set of Python source code files that provide a number of analysis and plotting utilities. The primary file is analysisUtils.py (108000 lines as of March 2021). The utilities were developed (and continue to be developed) for ALMA commissioning and data reduction and are, in many cases, also useful for EVLA data reduction. This CASA Guide documents some of the most useful functions contained in the analysisUtils Python module from a user's perspective, but most of the functions have fairly complete inline help. <br />
<br />
=Getting Started=<br />
<br />
'''If you are working on a machine at NRAO Charlottesville, then you can skip Step 1, because the latest modules are already available at: /users/thunter/AIV/science/analysis_scripts. ''' <br />
<br />
'''Step 1:''' Download Analysis Utilities <br />
There are two options:<br />
<br />
a) [ftp://ftp.cv.nrao.edu/pub/casaguides/analysis_scripts.tar NRAO ftp server]. (If your browser does not want to open an ftp link, then try right click to save the link and select the application to use to be wget.) <br />
<br />
b) [https://zenodo.org/record/7502160 zenodo] (this provides a citeable DOI: 10.5281/zenodo.7502160)<br />
<br />
Once downloaded, then extract the tar ball. From a Unix command line this can be done with<br />
<br />
<source lang="bash"><br />
$ tar xvf analysis_scripts.tar<br />
</source><br />
<br />
Seven files will be extracted -- a README file, and six python scripts. The README file contains the time<br />
and date that the tar ball was generated, which can be useful for reporting bugs. The [ftp://ftp.cv.nrao.edu/pub/casaguides/history history] file available at the ftp site contains a list of major changes in each version. <br />
<br />
'''Step 2:''' Edit your existing casa initialization file (in CASA5: <tt>$HOME/.casa/init.py</tt>, or in CASA6: <tt>$HOME$/.casa/startup.py</tt>) or create a new file if it does not already exist, and add the following<br />
<br />
<source lang="Python"><br />
import sys<br />
sys.path.append("/PATH_TO_ANALYSIS_SCRIPTS/")<br />
import analysisUtils as au<br />
</source><br />
<br />
where <tt>/PATH_TO_ANALYSIS_SCRIPTS/</tt> is the path to the directory you just extracted from the tar ball. An example can be downloaded and edited here: [[File:Init.py]]. When this is done, start casapy and you will have access to all the functions contained in the analysisUtils module.<br />
<br />
=Support=<br />
<br />
Once installed, functions can be searched for by name via minimum match string with the au.help function (where comma denotes a logical AND): e.g. au.help('antenna') or au.help('spw,caltable'). As in python, inline help for an individual function is accessed by typing 'help au.planet'. For further help with analysisUtils installation or usage, you may submit a helpdesk ticket to the Data Reduction category at https://help.almascience.org/. Be advised that analysisUtils is a commissioning tool, so the level of support available is not as extensive as for CASA, but we do welcome feedback. If your question is with regard to [[plotbandpass]], then please provide the version number of this program (it is printed to the screen at execution time and on the plots). In all cases, please provide the release date of your analysisUtils, which can be viewed from inside casapy as follows:<br />
<source lang="Python"><br />
au.version()<br />
</source><br />
<br />
=Script generators=<br />
The documentation for the two script generators almaqa2csg.py and almaqa2isg.py, as well as how to provide feedback or get support for their usage, can be found at this link: https://confluence.alma.cl/display/EAPR/ALMA+Data+Analysis+Utilities<br />
<br />
=Some Examples of Functions=<br />
<b>Up-to-date, online help is available for all of these functions (and more) at the CASA prompt by simply using the command: help(au.functionName).</b> To search for functions, you can type au.help('phrase'), where phrase is case-insensitive and minimum-match. Multiple phrases can be combined with a comma, for example au.help('baseline,stats').<br />
; [[angularSeparationOfFields]]<br />
: reads an ms and computes the angular separation of all fields, or a subset<br />
; [[buildConfigurationFile]]<br />
: reads the list of antenna stations in an .ms and creates a .cfg file suitable for [http://casa.nrao.edu/docs/taskref/simobserve-task.html simobserve]<br />
; [[editIntents]]<br />
: change the intents for a specified field in an ms (based on John Lightfoot's pipeline script)<br />
; [[getBaselineLengths]]<br />
: returns a dictionary of the baseline lengths in your ms, by default sorted by length<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/GetBaselineStats getBaselineStats]<br />
: returns a bunch of statistics on the baseline lengths in your ms<br />
; [[gjincBeam]]<br />
: compute the effective restoring beam obtained from the casa command {{sdimaging}} when using the GJINC gridding kernel<br />
; [[lstrange]]<br />
: lists the range of LST, UT, MJD for the whole ms, and for scans with OBSERVE_TARGET intent (including the elevation range)<br />
; [[obslist]]<br />
: prints the antenna station coordinates in local offsets from the Center of Array, and computes longest/shortest baselines<br />
; [[planet]]<br />
: contacts the JPL Horizons telnet service and returns the J2000 position, velocity, angular diameter, range and range rate of a solar system object<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/PlanetFlux planetFlux]<br />
: calls Bryan Butler's solar_system_setjy module in CASA to compute flux density vs. time or flux density vs. frequency <br />
; [[planetPlots]]<br />
: generates a plots of uv amplitude vs. uv distance for a grid of ALMA configurations and observing frequencies<br />
; [[plotbandpass]] <br />
: a faster version of plotcal for bandpass tables, with useful overlay capabilities. This function has been incorporated into CASA as a task.<br />
; [[plotSunDuringTrack]]<br />
: generates a plot of the Sun's az vs. el during an ms or ASDM<br />
; [[plotconfig]] <br />
: plot any standard telescope configuration of observatories known to casa, and return an array of the sorted baseline lengths<br />
; [[plotmosaic]]<br />
: shows relative location of pointings in an .ms<br />
; [[plotPWV]]<br />
: reads the PWV from each antenna's WVR from the ASDM_CALWVR table in an ms, and creates a plot vs. time<br />
; [[plotweather]]<br />
: plot weather conditions vs. time for your ms<br />
; [[plotWVRSolutions]]<br />
: plot the WVR solutions in terms of baseline-based phase corrections<br />
; [[spectralindex]]<br />
: performs a least-squares fit to the multi-spw output from {{fluxscale}}<br />
; [[timeOnSource]]<br />
: compute the total time spent integrating on-source for each specified field<br />
<br />
<b>A list of additional 1200+ functions, and their brief descriptions, can be found at [https://safe.nrao.edu/wiki/bin/view/Main/CasaExtensions Todd's wikipage]</b></div>Thunterhttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities&diff=34005Analysis Utilities2023-01-14T19:23:25Z<p>Thunter: /* Getting Started */</p>
<hr />
<div>[[Category:CASA]] [[Category:ALMA]] [[Category:EVLA]]<br />
Analysis Utilities (or analysisUtils for short) is a small set of Python source code files that provide a number of analysis and plotting utilities. The primary file is analysisUtils.py (108000 lines as of March 2021). The utilities were developed (and continue to be developed) for ALMA commissioning and data reduction and are, in many cases, also useful for EVLA data reduction. This CASA Guide documents some of the most useful functions contained in the analysisUtils Python module from a user's perspective, but most of the functions have fairly complete inline help. <br />
<br />
=Getting Started=<br />
<br />
'''If you are working on a machine at NRAO Charlottesville, then you can skip Step 1, because the latest modules are already available at: /users/thunter/AIV/science/analysis_scripts. ''' <br />
<br />
'''Step 1:''' Download Analysis Utilities <br />
There are two options:<br />
<br />
a) [ftp://ftp.cv.nrao.edu/pub/casaguides/analysis_scripts.tar NRAO ftp server]. (If your browser does not want to open an ftp link, then try right click to save the link and select the application to use to be wget.) <br />
<br />
b) [https://zenodo.org/record/7502160 zenodo] (this provides a citeable DOI: )<br />
<br />
Once downloaded, then extract the tar ball. From a Unix command line this can be done with<br />
<br />
<source lang="bash"><br />
$ tar xvf analysis_scripts.tar<br />
</source><br />
<br />
Seven files will be extracted -- a README file, and six python scripts. The README file contains the time<br />
and date that the tar ball was generated, which can be useful for reporting bugs. The [ftp://ftp.cv.nrao.edu/pub/casaguides/history history] file available at the ftp site contains a list of major changes in each version. <br />
<br />
'''Step 2:''' Edit your existing casa initialization file (in CASA5: <tt>$HOME/.casa/init.py</tt>, or in CASA6: <tt>$HOME$/.casa/startup.py</tt>) or create a new file if it does not already exist, and add the following<br />
<br />
<source lang="Python"><br />
import sys<br />
sys.path.append("/PATH_TO_ANALYSIS_SCRIPTS/")<br />
import analysisUtils as au<br />
</source><br />
<br />
where <tt>/PATH_TO_ANALYSIS_SCRIPTS/</tt> is the path to the directory you just extracted from the tar ball. An example can be downloaded and edited here: [[File:Init.py]]. When this is done, start casapy and you will have access to all the functions contained in the analysisUtils module.<br />
<br />
=Support=<br />
<br />
Once installed, functions can be searched for by name via minimum match string with the au.help function (where comma denotes a logical AND): e.g. au.help('antenna') or au.help('spw,caltable'). As in python, inline help for an individual function is accessed by typing 'help au.planet'. For further help with analysisUtils installation or usage, you may submit a helpdesk ticket to the Data Reduction category at https://help.almascience.org/. Be advised that analysisUtils is a commissioning tool, so the level of support available is not as extensive as for CASA, but we do welcome feedback. If your question is with regard to [[plotbandpass]], then please provide the version number of this program (it is printed to the screen at execution time and on the plots). In all cases, please provide the release date of your analysisUtils, which can be viewed from inside casapy as follows:<br />
<source lang="Python"><br />
au.version()<br />
</source><br />
<br />
=Script generators=<br />
The documentation for the two script generators almaqa2csg.py and almaqa2isg.py, as well as how to provide feedback or get support for their usage, can be found at this link: https://confluence.alma.cl/display/EAPR/ALMA+Data+Analysis+Utilities<br />
<br />
=Some Examples of Functions=<br />
<b>Up-to-date, online help is available for all of these functions (and more) at the CASA prompt by simply using the command: help(au.functionName).</b> To search for functions, you can type au.help('phrase'), where phrase is case-insensitive and minimum-match. Multiple phrases can be combined with a comma, for example au.help('baseline,stats').<br />
; [[angularSeparationOfFields]]<br />
: reads an ms and computes the angular separation of all fields, or a subset<br />
; [[buildConfigurationFile]]<br />
: reads the list of antenna stations in an .ms and creates a .cfg file suitable for [http://casa.nrao.edu/docs/taskref/simobserve-task.html simobserve]<br />
; [[editIntents]]<br />
: change the intents for a specified field in an ms (based on John Lightfoot's pipeline script)<br />
; [[getBaselineLengths]]<br />
: returns a dictionary of the baseline lengths in your ms, by default sorted by length<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/GetBaselineStats getBaselineStats]<br />
: returns a bunch of statistics on the baseline lengths in your ms<br />
; [[gjincBeam]]<br />
: compute the effective restoring beam obtained from the casa command {{sdimaging}} when using the GJINC gridding kernel<br />
; [[lstrange]]<br />
: lists the range of LST, UT, MJD for the whole ms, and for scans with OBSERVE_TARGET intent (including the elevation range)<br />
; [[obslist]]<br />
: prints the antenna station coordinates in local offsets from the Center of Array, and computes longest/shortest baselines<br />
; [[planet]]<br />
: contacts the JPL Horizons telnet service and returns the J2000 position, velocity, angular diameter, range and range rate of a solar system object<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/PlanetFlux planetFlux]<br />
: calls Bryan Butler's solar_system_setjy module in CASA to compute flux density vs. time or flux density vs. frequency <br />
; [[planetPlots]]<br />
: generates a plots of uv amplitude vs. uv distance for a grid of ALMA configurations and observing frequencies<br />
; [[plotbandpass]] <br />
: a faster version of plotcal for bandpass tables, with useful overlay capabilities. This function has been incorporated into CASA as a task.<br />
; [[plotSunDuringTrack]]<br />
: generates a plot of the Sun's az vs. el during an ms or ASDM<br />
; [[plotconfig]] <br />
: plot any standard telescope configuration of observatories known to casa, and return an array of the sorted baseline lengths<br />
; [[plotmosaic]]<br />
: shows relative location of pointings in an .ms<br />
; [[plotPWV]]<br />
: reads the PWV from each antenna's WVR from the ASDM_CALWVR table in an ms, and creates a plot vs. time<br />
; [[plotweather]]<br />
: plot weather conditions vs. time for your ms<br />
; [[plotWVRSolutions]]<br />
: plot the WVR solutions in terms of baseline-based phase corrections<br />
; [[spectralindex]]<br />
: performs a least-squares fit to the multi-spw output from {{fluxscale}}<br />
; [[timeOnSource]]<br />
: compute the total time spent integrating on-source for each specified field<br />
<br />
<b>A list of additional 1200+ functions, and their brief descriptions, can be found at [https://safe.nrao.edu/wiki/bin/view/Main/CasaExtensions Todd's wikipage]</b></div>Thunterhttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities&diff=34004Analysis Utilities2023-01-14T19:23:01Z<p>Thunter: /* Getting Started */</p>
<hr />
<div>[[Category:CASA]] [[Category:ALMA]] [[Category:EVLA]]<br />
Analysis Utilities (or analysisUtils for short) is a small set of Python source code files that provide a number of analysis and plotting utilities. The primary file is analysisUtils.py (108000 lines as of March 2021). The utilities were developed (and continue to be developed) for ALMA commissioning and data reduction and are, in many cases, also useful for EVLA data reduction. This CASA Guide documents some of the most useful functions contained in the analysisUtils Python module from a user's perspective, but most of the functions have fairly complete inline help. <br />
<br />
=Getting Started=<br />
<br />
'''If you are working on a machine at NRAO Charlottesville, then you can skip Step 1, because the latest modules are already available at: /users/thunter/AIV/science/analysis_scripts. ''' <br />
<br />
'''Step 1:''' Download Analysis Utilities <br />
There are two options:<br />
<br />
a) [ftp://ftp.cv.nrao.edu/pub/casaguides/analysis_scripts.tar NRAO ftp server]. If your browser does not want to open an ftp link, then try right click to save the link and select the application to use to be wget. <br />
<br />
b) [https://zenodo.org/record/7502160 zenodo]<br />
<br />
Once downloaded, then extract the tar ball. From a Unix command line this can be done with<br />
<br />
<source lang="bash"><br />
$ tar xvf analysis_scripts.tar<br />
</source><br />
<br />
Seven files will be extracted -- a README file, and six python scripts. The README file contains the time<br />
and date that the tar ball was generated, which can be useful for reporting bugs. The [ftp://ftp.cv.nrao.edu/pub/casaguides/history history] file available at the ftp site contains a list of major changes in each version. <br />
<br />
'''Step 2:''' Edit your existing casa initialization file (in CASA5: <tt>$HOME/.casa/init.py</tt>, or in CASA6: <tt>$HOME$/.casa/startup.py</tt>) or create a new file if it does not already exist, and add the following<br />
<br />
<source lang="Python"><br />
import sys<br />
sys.path.append("/PATH_TO_ANALYSIS_SCRIPTS/")<br />
import analysisUtils as au<br />
</source><br />
<br />
where <tt>/PATH_TO_ANALYSIS_SCRIPTS/</tt> is the path to the directory you just extracted from the tar ball. An example can be downloaded and edited here: [[File:Init.py]]. When this is done, start casapy and you will have access to all the functions contained in the analysisUtils module.<br />
<br />
=Support=<br />
<br />
Once installed, functions can be searched for by name via minimum match string with the au.help function (where comma denotes a logical AND): e.g. au.help('antenna') or au.help('spw,caltable'). As in python, inline help for an individual function is accessed by typing 'help au.planet'. For further help with analysisUtils installation or usage, you may submit a helpdesk ticket to the Data Reduction category at https://help.almascience.org/. Be advised that analysisUtils is a commissioning tool, so the level of support available is not as extensive as for CASA, but we do welcome feedback. If your question is with regard to [[plotbandpass]], then please provide the version number of this program (it is printed to the screen at execution time and on the plots). In all cases, please provide the release date of your analysisUtils, which can be viewed from inside casapy as follows:<br />
<source lang="Python"><br />
au.version()<br />
</source><br />
<br />
=Script generators=<br />
The documentation for the two script generators almaqa2csg.py and almaqa2isg.py, as well as how to provide feedback or get support for their usage, can be found at this link: https://confluence.alma.cl/display/EAPR/ALMA+Data+Analysis+Utilities<br />
<br />
=Some Examples of Functions=<br />
<b>Up-to-date, online help is available for all of these functions (and more) at the CASA prompt by simply using the command: help(au.functionName).</b> To search for functions, you can type au.help('phrase'), where phrase is case-insensitive and minimum-match. Multiple phrases can be combined with a comma, for example au.help('baseline,stats').<br />
; [[angularSeparationOfFields]]<br />
: reads an ms and computes the angular separation of all fields, or a subset<br />
; [[buildConfigurationFile]]<br />
: reads the list of antenna stations in an .ms and creates a .cfg file suitable for [http://casa.nrao.edu/docs/taskref/simobserve-task.html simobserve]<br />
; [[editIntents]]<br />
: change the intents for a specified field in an ms (based on John Lightfoot's pipeline script)<br />
; [[getBaselineLengths]]<br />
: returns a dictionary of the baseline lengths in your ms, by default sorted by length<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/GetBaselineStats getBaselineStats]<br />
: returns a bunch of statistics on the baseline lengths in your ms<br />
; [[gjincBeam]]<br />
: compute the effective restoring beam obtained from the casa command {{sdimaging}} when using the GJINC gridding kernel<br />
; [[lstrange]]<br />
: lists the range of LST, UT, MJD for the whole ms, and for scans with OBSERVE_TARGET intent (including the elevation range)<br />
; [[obslist]]<br />
: prints the antenna station coordinates in local offsets from the Center of Array, and computes longest/shortest baselines<br />
; [[planet]]<br />
: contacts the JPL Horizons telnet service and returns the J2000 position, velocity, angular diameter, range and range rate of a solar system object<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/PlanetFlux planetFlux]<br />
: calls Bryan Butler's solar_system_setjy module in CASA to compute flux density vs. time or flux density vs. frequency <br />
; [[planetPlots]]<br />
: generates a plots of uv amplitude vs. uv distance for a grid of ALMA configurations and observing frequencies<br />
; [[plotbandpass]] <br />
: a faster version of plotcal for bandpass tables, with useful overlay capabilities. This function has been incorporated into CASA as a task.<br />
; [[plotSunDuringTrack]]<br />
: generates a plot of the Sun's az vs. el during an ms or ASDM<br />
; [[plotconfig]] <br />
: plot any standard telescope configuration of observatories known to casa, and return an array of the sorted baseline lengths<br />
; [[plotmosaic]]<br />
: shows relative location of pointings in an .ms<br />
; [[plotPWV]]<br />
: reads the PWV from each antenna's WVR from the ASDM_CALWVR table in an ms, and creates a plot vs. time<br />
; [[plotweather]]<br />
: plot weather conditions vs. time for your ms<br />
; [[plotWVRSolutions]]<br />
: plot the WVR solutions in terms of baseline-based phase corrections<br />
; [[spectralindex]]<br />
: performs a least-squares fit to the multi-spw output from {{fluxscale}}<br />
; [[timeOnSource]]<br />
: compute the total time spent integrating on-source for each specified field<br />
<br />
<b>A list of additional 1200+ functions, and their brief descriptions, can be found at [https://safe.nrao.edu/wiki/bin/view/Main/CasaExtensions Todd's wikipage]</b></div>Thunterhttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities&diff=34003Analysis Utilities2023-01-14T19:22:47Z<p>Thunter: /* Getting Started */</p>
<hr />
<div>[[Category:CASA]] [[Category:ALMA]] [[Category:EVLA]]<br />
Analysis Utilities (or analysisUtils for short) is a small set of Python source code files that provide a number of analysis and plotting utilities. The primary file is analysisUtils.py (108000 lines as of March 2021). The utilities were developed (and continue to be developed) for ALMA commissioning and data reduction and are, in many cases, also useful for EVLA data reduction. This CASA Guide documents some of the most useful functions contained in the analysisUtils Python module from a user's perspective, but most of the functions have fairly complete inline help. <br />
<br />
=Getting Started=<br />
<br />
'''If you are working on a machine at NRAO Charlottesville, then you can skip Step 1, because the latest modules are already available at: /users/thunter/AIV/science/analysis_scripts. ''' <br />
<br />
'''Step 1:''' Download Analysis Utilities <br />
There are two options:<br />
<br />
a) [ftp://ftp.cv.nrao.edu/pub/casaguides/analysis_scripts.tar NRAO]. If your browser does not want to open an ftp link, then try right click to save the link and select the application to use to be wget. <br />
<br />
b) [https://zenodo.org/record/7502160 zenodo]<br />
<br />
Once downloaded, then extract the tar ball. From a Unix command line this can be done with<br />
<br />
<source lang="bash"><br />
$ tar xvf analysis_scripts.tar<br />
</source><br />
<br />
Seven files will be extracted -- a README file, and six python scripts. The README file contains the time<br />
and date that the tar ball was generated, which can be useful for reporting bugs. The [ftp://ftp.cv.nrao.edu/pub/casaguides/history history] file available at the ftp site contains a list of major changes in each version. <br />
<br />
'''Step 2:''' Edit your existing casa initialization file (in CASA5: <tt>$HOME/.casa/init.py</tt>, or in CASA6: <tt>$HOME$/.casa/startup.py</tt>) or create a new file if it does not already exist, and add the following<br />
<br />
<source lang="Python"><br />
import sys<br />
sys.path.append("/PATH_TO_ANALYSIS_SCRIPTS/")<br />
import analysisUtils as au<br />
</source><br />
<br />
where <tt>/PATH_TO_ANALYSIS_SCRIPTS/</tt> is the path to the directory you just extracted from the tar ball. An example can be downloaded and edited here: [[File:Init.py]]. When this is done, start casapy and you will have access to all the functions contained in the analysisUtils module.<br />
<br />
=Support=<br />
<br />
Once installed, functions can be searched for by name via minimum match string with the au.help function (where comma denotes a logical AND): e.g. au.help('antenna') or au.help('spw,caltable'). As in python, inline help for an individual function is accessed by typing 'help au.planet'. For further help with analysisUtils installation or usage, you may submit a helpdesk ticket to the Data Reduction category at https://help.almascience.org/. Be advised that analysisUtils is a commissioning tool, so the level of support available is not as extensive as for CASA, but we do welcome feedback. If your question is with regard to [[plotbandpass]], then please provide the version number of this program (it is printed to the screen at execution time and on the plots). In all cases, please provide the release date of your analysisUtils, which can be viewed from inside casapy as follows:<br />
<source lang="Python"><br />
au.version()<br />
</source><br />
<br />
=Script generators=<br />
The documentation for the two script generators almaqa2csg.py and almaqa2isg.py, as well as how to provide feedback or get support for their usage, can be found at this link: https://confluence.alma.cl/display/EAPR/ALMA+Data+Analysis+Utilities<br />
<br />
=Some Examples of Functions=<br />
<b>Up-to-date, online help is available for all of these functions (and more) at the CASA prompt by simply using the command: help(au.functionName).</b> To search for functions, you can type au.help('phrase'), where phrase is case-insensitive and minimum-match. Multiple phrases can be combined with a comma, for example au.help('baseline,stats').<br />
; [[angularSeparationOfFields]]<br />
: reads an ms and computes the angular separation of all fields, or a subset<br />
; [[buildConfigurationFile]]<br />
: reads the list of antenna stations in an .ms and creates a .cfg file suitable for [http://casa.nrao.edu/docs/taskref/simobserve-task.html simobserve]<br />
; [[editIntents]]<br />
: change the intents for a specified field in an ms (based on John Lightfoot's pipeline script)<br />
; [[getBaselineLengths]]<br />
: returns a dictionary of the baseline lengths in your ms, by default sorted by length<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/GetBaselineStats getBaselineStats]<br />
: returns a bunch of statistics on the baseline lengths in your ms<br />
; [[gjincBeam]]<br />
: compute the effective restoring beam obtained from the casa command {{sdimaging}} when using the GJINC gridding kernel<br />
; [[lstrange]]<br />
: lists the range of LST, UT, MJD for the whole ms, and for scans with OBSERVE_TARGET intent (including the elevation range)<br />
; [[obslist]]<br />
: prints the antenna station coordinates in local offsets from the Center of Array, and computes longest/shortest baselines<br />
; [[planet]]<br />
: contacts the JPL Horizons telnet service and returns the J2000 position, velocity, angular diameter, range and range rate of a solar system object<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/PlanetFlux planetFlux]<br />
: calls Bryan Butler's solar_system_setjy module in CASA to compute flux density vs. time or flux density vs. frequency <br />
; [[planetPlots]]<br />
: generates a plots of uv amplitude vs. uv distance for a grid of ALMA configurations and observing frequencies<br />
; [[plotbandpass]] <br />
: a faster version of plotcal for bandpass tables, with useful overlay capabilities. This function has been incorporated into CASA as a task.<br />
; [[plotSunDuringTrack]]<br />
: generates a plot of the Sun's az vs. el during an ms or ASDM<br />
; [[plotconfig]] <br />
: plot any standard telescope configuration of observatories known to casa, and return an array of the sorted baseline lengths<br />
; [[plotmosaic]]<br />
: shows relative location of pointings in an .ms<br />
; [[plotPWV]]<br />
: reads the PWV from each antenna's WVR from the ASDM_CALWVR table in an ms, and creates a plot vs. time<br />
; [[plotweather]]<br />
: plot weather conditions vs. time for your ms<br />
; [[plotWVRSolutions]]<br />
: plot the WVR solutions in terms of baseline-based phase corrections<br />
; [[spectralindex]]<br />
: performs a least-squares fit to the multi-spw output from {{fluxscale}}<br />
; [[timeOnSource]]<br />
: compute the total time spent integrating on-source for each specified field<br />
<br />
<b>A list of additional 1200+ functions, and their brief descriptions, can be found at [https://safe.nrao.edu/wiki/bin/view/Main/CasaExtensions Todd's wikipage]</b></div>Thunterhttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities&diff=34002Analysis Utilities2023-01-14T19:22:18Z<p>Thunter: /* Getting Started */</p>
<hr />
<div>[[Category:CASA]] [[Category:ALMA]] [[Category:EVLA]]<br />
Analysis Utilities (or analysisUtils for short) is a small set of Python source code files that provide a number of analysis and plotting utilities. The primary file is analysisUtils.py (108000 lines as of March 2021). The utilities were developed (and continue to be developed) for ALMA commissioning and data reduction and are, in many cases, also useful for EVLA data reduction. This CASA Guide documents some of the most useful functions contained in the analysisUtils Python module from a user's perspective, but most of the functions have fairly complete inline help. <br />
<br />
=Getting Started=<br />
<br />
'''If you are working on a machine at NRAO Charlottesville, then you can skip Step 1, because the latest modules are already available at: /users/thunter/AIV/science/analysis_scripts. ''' <br />
<br />
'''Step 1:''' Download Analysis Utilities from either:<br />
a) [ftp://ftp.cv.nrao.edu/pub/casaguides/analysis_scripts.tar NRAO]. If your browser does not want to open an ftp link, then try right click to save the link and select the application to use to be wget. <br />
b) [https://zenodo.org/record/7502160 zenodo]<br />
<br />
Once downloaded, then extract the tar ball. From a Unix command line this can be done with<br />
<br />
<source lang="bash"><br />
$ tar xvf analysis_scripts.tar<br />
</source><br />
<br />
Seven files will be extracted -- a README file, and six python scripts. The README file contains the time<br />
and date that the tar ball was generated, which can be useful for reporting bugs. The [ftp://ftp.cv.nrao.edu/pub/casaguides/history history] file available at the ftp site contains a list of major changes in each version. <br />
<br />
'''Step 2:''' Edit your existing casa initialization file (in CASA5: <tt>$HOME/.casa/init.py</tt>, or in CASA6: <tt>$HOME$/.casa/startup.py</tt>) or create a new file if it does not already exist, and add the following<br />
<br />
<source lang="Python"><br />
import sys<br />
sys.path.append("/PATH_TO_ANALYSIS_SCRIPTS/")<br />
import analysisUtils as au<br />
</source><br />
<br />
where <tt>/PATH_TO_ANALYSIS_SCRIPTS/</tt> is the path to the directory you just extracted from the tar ball. An example can be downloaded and edited here: [[File:Init.py]]. When this is done, start casapy and you will have access to all the functions contained in the analysisUtils module.<br />
<br />
=Support=<br />
<br />
Once installed, functions can be searched for by name via minimum match string with the au.help function (where comma denotes a logical AND): e.g. au.help('antenna') or au.help('spw,caltable'). As in python, inline help for an individual function is accessed by typing 'help au.planet'. For further help with analysisUtils installation or usage, you may submit a helpdesk ticket to the Data Reduction category at https://help.almascience.org/. Be advised that analysisUtils is a commissioning tool, so the level of support available is not as extensive as for CASA, but we do welcome feedback. If your question is with regard to [[plotbandpass]], then please provide the version number of this program (it is printed to the screen at execution time and on the plots). In all cases, please provide the release date of your analysisUtils, which can be viewed from inside casapy as follows:<br />
<source lang="Python"><br />
au.version()<br />
</source><br />
<br />
=Script generators=<br />
The documentation for the two script generators almaqa2csg.py and almaqa2isg.py, as well as how to provide feedback or get support for their usage, can be found at this link: https://confluence.alma.cl/display/EAPR/ALMA+Data+Analysis+Utilities<br />
<br />
=Some Examples of Functions=<br />
<b>Up-to-date, online help is available for all of these functions (and more) at the CASA prompt by simply using the command: help(au.functionName).</b> To search for functions, you can type au.help('phrase'), where phrase is case-insensitive and minimum-match. Multiple phrases can be combined with a comma, for example au.help('baseline,stats').<br />
; [[angularSeparationOfFields]]<br />
: reads an ms and computes the angular separation of all fields, or a subset<br />
; [[buildConfigurationFile]]<br />
: reads the list of antenna stations in an .ms and creates a .cfg file suitable for [http://casa.nrao.edu/docs/taskref/simobserve-task.html simobserve]<br />
; [[editIntents]]<br />
: change the intents for a specified field in an ms (based on John Lightfoot's pipeline script)<br />
; [[getBaselineLengths]]<br />
: returns a dictionary of the baseline lengths in your ms, by default sorted by length<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/GetBaselineStats getBaselineStats]<br />
: returns a bunch of statistics on the baseline lengths in your ms<br />
; [[gjincBeam]]<br />
: compute the effective restoring beam obtained from the casa command {{sdimaging}} when using the GJINC gridding kernel<br />
; [[lstrange]]<br />
: lists the range of LST, UT, MJD for the whole ms, and for scans with OBSERVE_TARGET intent (including the elevation range)<br />
; [[obslist]]<br />
: prints the antenna station coordinates in local offsets from the Center of Array, and computes longest/shortest baselines<br />
; [[planet]]<br />
: contacts the JPL Horizons telnet service and returns the J2000 position, velocity, angular diameter, range and range rate of a solar system object<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/PlanetFlux planetFlux]<br />
: calls Bryan Butler's solar_system_setjy module in CASA to compute flux density vs. time or flux density vs. frequency <br />
; [[planetPlots]]<br />
: generates a plots of uv amplitude vs. uv distance for a grid of ALMA configurations and observing frequencies<br />
; [[plotbandpass]] <br />
: a faster version of plotcal for bandpass tables, with useful overlay capabilities. This function has been incorporated into CASA as a task.<br />
; [[plotSunDuringTrack]]<br />
: generates a plot of the Sun's az vs. el during an ms or ASDM<br />
; [[plotconfig]] <br />
: plot any standard telescope configuration of observatories known to casa, and return an array of the sorted baseline lengths<br />
; [[plotmosaic]]<br />
: shows relative location of pointings in an .ms<br />
; [[plotPWV]]<br />
: reads the PWV from each antenna's WVR from the ASDM_CALWVR table in an ms, and creates a plot vs. time<br />
; [[plotweather]]<br />
: plot weather conditions vs. time for your ms<br />
; [[plotWVRSolutions]]<br />
: plot the WVR solutions in terms of baseline-based phase corrections<br />
; [[spectralindex]]<br />
: performs a least-squares fit to the multi-spw output from {{fluxscale}}<br />
; [[timeOnSource]]<br />
: compute the total time spent integrating on-source for each specified field<br />
<br />
<b>A list of additional 1200+ functions, and their brief descriptions, can be found at [https://safe.nrao.edu/wiki/bin/view/Main/CasaExtensions Todd's wikipage]</b></div>Thunterhttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities&diff=34001Analysis Utilities2023-01-14T19:21:45Z<p>Thunter: /* Getting Started */</p>
<hr />
<div>[[Category:CASA]] [[Category:ALMA]] [[Category:EVLA]]<br />
Analysis Utilities (or analysisUtils for short) is a small set of Python source code files that provide a number of analysis and plotting utilities. The primary file is analysisUtils.py (108000 lines as of March 2021). The utilities were developed (and continue to be developed) for ALMA commissioning and data reduction and are, in many cases, also useful for EVLA data reduction. This CASA Guide documents some of the most useful functions contained in the analysisUtils Python module from a user's perspective, but most of the functions have fairly complete inline help. <br />
<br />
=Getting Started=<br />
<br />
'''If you are working on a machine at NRAO Charlottesville, then you can skip Step 1, because the latest modules are already available at: /users/thunter/AIV/science/analysis_scripts. ''' <br />
<br />
'''Step 1:''' Download Analysis Utilities from either:<br />
a) [wget://ftp.cv.nrao.edu/pub/casaguides/analysis_scripts.tar NRAO]. If your browser does not want to open an ftp link, then try right click to save the link and select the application to use to wget. <br />
b) [https://zenodo.org/record/7502160 zenodo]<br />
<br />
Once downloaded, then extract the tar ball. From a Unix command line this can be done with<br />
<br />
<source lang="bash"><br />
$ tar xvf analysis_scripts.tar<br />
</source><br />
<br />
Seven files will be extracted -- a README file, and six python scripts. The README file contains the time<br />
and date that the tar ball was generated, which can be useful for reporting bugs. The [ftp://ftp.cv.nrao.edu/pub/casaguides/history history] file available at the ftp site contains a list of major changes in each version. <br />
<br />
'''Step 2:''' Edit your existing casa initialization file (in CASA5: <tt>$HOME/.casa/init.py</tt>, or in CASA6: <tt>$HOME$/.casa/startup.py</tt>) or create a new file if it does not already exist, and add the following<br />
<br />
<source lang="Python"><br />
import sys<br />
sys.path.append("/PATH_TO_ANALYSIS_SCRIPTS/")<br />
import analysisUtils as au<br />
</source><br />
<br />
where <tt>/PATH_TO_ANALYSIS_SCRIPTS/</tt> is the path to the directory you just extracted from the tar ball. An example can be downloaded and edited here: [[File:Init.py]]. When this is done, start casapy and you will have access to all the functions contained in the analysisUtils module.<br />
<br />
=Support=<br />
<br />
Once installed, functions can be searched for by name via minimum match string with the au.help function (where comma denotes a logical AND): e.g. au.help('antenna') or au.help('spw,caltable'). As in python, inline help for an individual function is accessed by typing 'help au.planet'. For further help with analysisUtils installation or usage, you may submit a helpdesk ticket to the Data Reduction category at https://help.almascience.org/. Be advised that analysisUtils is a commissioning tool, so the level of support available is not as extensive as for CASA, but we do welcome feedback. If your question is with regard to [[plotbandpass]], then please provide the version number of this program (it is printed to the screen at execution time and on the plots). In all cases, please provide the release date of your analysisUtils, which can be viewed from inside casapy as follows:<br />
<source lang="Python"><br />
au.version()<br />
</source><br />
<br />
=Script generators=<br />
The documentation for the two script generators almaqa2csg.py and almaqa2isg.py, as well as how to provide feedback or get support for their usage, can be found at this link: https://confluence.alma.cl/display/EAPR/ALMA+Data+Analysis+Utilities<br />
<br />
=Some Examples of Functions=<br />
<b>Up-to-date, online help is available for all of these functions (and more) at the CASA prompt by simply using the command: help(au.functionName).</b> To search for functions, you can type au.help('phrase'), where phrase is case-insensitive and minimum-match. Multiple phrases can be combined with a comma, for example au.help('baseline,stats').<br />
; [[angularSeparationOfFields]]<br />
: reads an ms and computes the angular separation of all fields, or a subset<br />
; [[buildConfigurationFile]]<br />
: reads the list of antenna stations in an .ms and creates a .cfg file suitable for [http://casa.nrao.edu/docs/taskref/simobserve-task.html simobserve]<br />
; [[editIntents]]<br />
: change the intents for a specified field in an ms (based on John Lightfoot's pipeline script)<br />
; [[getBaselineLengths]]<br />
: returns a dictionary of the baseline lengths in your ms, by default sorted by length<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/GetBaselineStats getBaselineStats]<br />
: returns a bunch of statistics on the baseline lengths in your ms<br />
; [[gjincBeam]]<br />
: compute the effective restoring beam obtained from the casa command {{sdimaging}} when using the GJINC gridding kernel<br />
; [[lstrange]]<br />
: lists the range of LST, UT, MJD for the whole ms, and for scans with OBSERVE_TARGET intent (including the elevation range)<br />
; [[obslist]]<br />
: prints the antenna station coordinates in local offsets from the Center of Array, and computes longest/shortest baselines<br />
; [[planet]]<br />
: contacts the JPL Horizons telnet service and returns the J2000 position, velocity, angular diameter, range and range rate of a solar system object<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/PlanetFlux planetFlux]<br />
: calls Bryan Butler's solar_system_setjy module in CASA to compute flux density vs. time or flux density vs. frequency <br />
; [[planetPlots]]<br />
: generates a plots of uv amplitude vs. uv distance for a grid of ALMA configurations and observing frequencies<br />
; [[plotbandpass]] <br />
: a faster version of plotcal for bandpass tables, with useful overlay capabilities. This function has been incorporated into CASA as a task.<br />
; [[plotSunDuringTrack]]<br />
: generates a plot of the Sun's az vs. el during an ms or ASDM<br />
; [[plotconfig]] <br />
: plot any standard telescope configuration of observatories known to casa, and return an array of the sorted baseline lengths<br />
; [[plotmosaic]]<br />
: shows relative location of pointings in an .ms<br />
; [[plotPWV]]<br />
: reads the PWV from each antenna's WVR from the ASDM_CALWVR table in an ms, and creates a plot vs. time<br />
; [[plotweather]]<br />
: plot weather conditions vs. time for your ms<br />
; [[plotWVRSolutions]]<br />
: plot the WVR solutions in terms of baseline-based phase corrections<br />
; [[spectralindex]]<br />
: performs a least-squares fit to the multi-spw output from {{fluxscale}}<br />
; [[timeOnSource]]<br />
: compute the total time spent integrating on-source for each specified field<br />
<br />
<b>A list of additional 1200+ functions, and their brief descriptions, can be found at [https://safe.nrao.edu/wiki/bin/view/Main/CasaExtensions Todd's wikipage]</b></div>Thunterhttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities&diff=34000Analysis Utilities2023-01-14T19:20:52Z<p>Thunter: /* Getting Started */</p>
<hr />
<div>[[Category:CASA]] [[Category:ALMA]] [[Category:EVLA]]<br />
Analysis Utilities (or analysisUtils for short) is a small set of Python source code files that provide a number of analysis and plotting utilities. The primary file is analysisUtils.py (108000 lines as of March 2021). The utilities were developed (and continue to be developed) for ALMA commissioning and data reduction and are, in many cases, also useful for EVLA data reduction. This CASA Guide documents some of the most useful functions contained in the analysisUtils Python module from a user's perspective, but most of the functions have fairly complete inline help. <br />
<br />
=Getting Started=<br />
<br />
'''If you are working on a machine at NRAO Charlottesville, then you can skip Step 1, because the latest modules are already available at: /users/thunter/AIV/science/analysis_scripts. ''' <br />
<br />
'''Step 1:''' Download Analysis Utilities from either:<br />
a) [wget://ftp.cv.nrao.edu/pub/casaguides/analysis_scripts.tar]. If your browser does not want to open an ftp link, then try right click to save the link and select the application to use to wget. <br />
b) [https://zenodo.org/record/7502160]<br />
<br />
Once downloaded, then extract the tar ball. From a Unix command line this can be done with<br />
<br />
<source lang="bash"><br />
$ tar xvf analysis_scripts.tar<br />
</source><br />
<br />
Seven files will be extracted -- a README file, and six python scripts. The README file contains the time<br />
and date that the tar ball was generated, which can be useful for reporting bugs. The [ftp://ftp.cv.nrao.edu/pub/casaguides/history history] file available at the ftp site contains a list of major changes in each version. <br />
<br />
'''Step 2:''' Edit your existing casa initialization file (in CASA5: <tt>$HOME/.casa/init.py</tt>, or in CASA6: <tt>$HOME$/.casa/startup.py</tt>) or create a new file if it does not already exist, and add the following<br />
<br />
<source lang="Python"><br />
import sys<br />
sys.path.append("/PATH_TO_ANALYSIS_SCRIPTS/")<br />
import analysisUtils as au<br />
</source><br />
<br />
where <tt>/PATH_TO_ANALYSIS_SCRIPTS/</tt> is the path to the directory you just extracted from the tar ball. An example can be downloaded and edited here: [[File:Init.py]]. When this is done, start casapy and you will have access to all the functions contained in the analysisUtils module.<br />
<br />
=Support=<br />
<br />
Once installed, functions can be searched for by name via minimum match string with the au.help function (where comma denotes a logical AND): e.g. au.help('antenna') or au.help('spw,caltable'). As in python, inline help for an individual function is accessed by typing 'help au.planet'. For further help with analysisUtils installation or usage, you may submit a helpdesk ticket to the Data Reduction category at https://help.almascience.org/. Be advised that analysisUtils is a commissioning tool, so the level of support available is not as extensive as for CASA, but we do welcome feedback. If your question is with regard to [[plotbandpass]], then please provide the version number of this program (it is printed to the screen at execution time and on the plots). In all cases, please provide the release date of your analysisUtils, which can be viewed from inside casapy as follows:<br />
<source lang="Python"><br />
au.version()<br />
</source><br />
<br />
=Script generators=<br />
The documentation for the two script generators almaqa2csg.py and almaqa2isg.py, as well as how to provide feedback or get support for their usage, can be found at this link: https://confluence.alma.cl/display/EAPR/ALMA+Data+Analysis+Utilities<br />
<br />
=Some Examples of Functions=<br />
<b>Up-to-date, online help is available for all of these functions (and more) at the CASA prompt by simply using the command: help(au.functionName).</b> To search for functions, you can type au.help('phrase'), where phrase is case-insensitive and minimum-match. Multiple phrases can be combined with a comma, for example au.help('baseline,stats').<br />
; [[angularSeparationOfFields]]<br />
: reads an ms and computes the angular separation of all fields, or a subset<br />
; [[buildConfigurationFile]]<br />
: reads the list of antenna stations in an .ms and creates a .cfg file suitable for [http://casa.nrao.edu/docs/taskref/simobserve-task.html simobserve]<br />
; [[editIntents]]<br />
: change the intents for a specified field in an ms (based on John Lightfoot's pipeline script)<br />
; [[getBaselineLengths]]<br />
: returns a dictionary of the baseline lengths in your ms, by default sorted by length<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/GetBaselineStats getBaselineStats]<br />
: returns a bunch of statistics on the baseline lengths in your ms<br />
; [[gjincBeam]]<br />
: compute the effective restoring beam obtained from the casa command {{sdimaging}} when using the GJINC gridding kernel<br />
; [[lstrange]]<br />
: lists the range of LST, UT, MJD for the whole ms, and for scans with OBSERVE_TARGET intent (including the elevation range)<br />
; [[obslist]]<br />
: prints the antenna station coordinates in local offsets from the Center of Array, and computes longest/shortest baselines<br />
; [[planet]]<br />
: contacts the JPL Horizons telnet service and returns the J2000 position, velocity, angular diameter, range and range rate of a solar system object<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/PlanetFlux planetFlux]<br />
: calls Bryan Butler's solar_system_setjy module in CASA to compute flux density vs. time or flux density vs. frequency <br />
; [[planetPlots]]<br />
: generates a plots of uv amplitude vs. uv distance for a grid of ALMA configurations and observing frequencies<br />
; [[plotbandpass]] <br />
: a faster version of plotcal for bandpass tables, with useful overlay capabilities. This function has been incorporated into CASA as a task.<br />
; [[plotSunDuringTrack]]<br />
: generates a plot of the Sun's az vs. el during an ms or ASDM<br />
; [[plotconfig]] <br />
: plot any standard telescope configuration of observatories known to casa, and return an array of the sorted baseline lengths<br />
; [[plotmosaic]]<br />
: shows relative location of pointings in an .ms<br />
; [[plotPWV]]<br />
: reads the PWV from each antenna's WVR from the ASDM_CALWVR table in an ms, and creates a plot vs. time<br />
; [[plotweather]]<br />
: plot weather conditions vs. time for your ms<br />
; [[plotWVRSolutions]]<br />
: plot the WVR solutions in terms of baseline-based phase corrections<br />
; [[spectralindex]]<br />
: performs a least-squares fit to the multi-spw output from {{fluxscale}}<br />
; [[timeOnSource]]<br />
: compute the total time spent integrating on-source for each specified field<br />
<br />
<b>A list of additional 1200+ functions, and their brief descriptions, can be found at [https://safe.nrao.edu/wiki/bin/view/Main/CasaExtensions Todd's wikipage]</b></div>Thunterhttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities&diff=33798Analysis Utilities2023-01-03T18:57:50Z<p>Thunter: </p>
<hr />
<div>[[Category:CASA]] [[Category:ALMA]] [[Category:EVLA]]<br />
Analysis Utilities (or analysisUtils for short) is a small set of Python source code files that provide a number of analysis and plotting utilities. The primary file is analysisUtils.py (108000 lines as of March 2021). The utilities were developed (and continue to be developed) for ALMA commissioning and data reduction and are, in many cases, also useful for EVLA data reduction. This CASA Guide documents some of the most useful functions contained in the analysisUtils Python module from a user's perspective, but most of the functions have fairly complete inline help. <br />
<br />
=Getting Started=<br />
<br />
'''If you are working on a machine at NRAO Charlottesville, then you can skip Step 1, because the latest modules are already available at: /users/thunter/AIV/science/analysis_scripts. ''' <br />
<br />
'''Step 1:''' Download Analysis Utilities from [ftp://ftp.cv.nrao.edu/pub/casaguides/analysis_scripts.tar here]. If your browser does not want to open an ftp link, then try right click to save the link and select the application to use to wget. Once downloaded, then extract the tar ball. From a Unix command line this can be done with<br />
<br />
<source lang="bash"><br />
$ tar xvf analysis_scripts.tar<br />
</source><br />
<br />
Seven files will be extracted -- a README file, and six python scripts. The README file contains the time<br />
and date that the tar ball was generated, which can be useful for reporting bugs. The [ftp://ftp.cv.nrao.edu/pub/casaguides/history history] file available at the ftp site contains a list of major changes in each version. <br />
<br />
'''Step 2:''' Edit your existing casa initialization file (in CASA5: <tt>$HOME/.casa/init.py</tt>, or in CASA6: <tt>$HOME$/.casa/startup.py</tt>) or create a new file if it does not already exist, and add the following<br />
<br />
<source lang="Python"><br />
import sys<br />
sys.path.append("/PATH_TO_ANALYSIS_SCRIPTS/")<br />
import analysisUtils as au<br />
</source><br />
<br />
where <tt>/PATH_TO_ANALYSIS_SCRIPTS/</tt> is the path to the directory you just extracted from the tar ball. An example can be downloaded and edited here: [[File:Init.py]]. When this is done, start casapy and you will have access to all the functions contained in the analysisUtils module.<br />
<br />
=Support=<br />
<br />
Once installed, functions can be searched for by name via minimum match string with the au.help function (where comma denotes a logical AND): e.g. au.help('antenna') or au.help('spw,caltable'). As in python, inline help for an individual function is accessed by typing 'help au.planet'. For further help with analysisUtils installation or usage, you may submit a helpdesk ticket to the Data Reduction category at https://help.almascience.org/. Be advised that analysisUtils is a commissioning tool, so the level of support available is not as extensive as for CASA, but we do welcome feedback. If your question is with regard to [[plotbandpass]], then please provide the version number of this program (it is printed to the screen at execution time and on the plots). In all cases, please provide the release date of your analysisUtils, which can be viewed from inside casapy as follows:<br />
<source lang="Python"><br />
au.version()<br />
</source><br />
<br />
=Script generators=<br />
The documentation for the two script generators almaqa2csg.py and almaqa2isg.py, as well as how to provide feedback or get support for their usage, can be found at this link: https://confluence.alma.cl/display/EAPR/ALMA+Data+Analysis+Utilities<br />
<br />
=Some Examples of Functions=<br />
<b>Up-to-date, online help is available for all of these functions (and more) at the CASA prompt by simply using the command: help(au.functionName).</b> To search for functions, you can type au.help('phrase'), where phrase is case-insensitive and minimum-match. Multiple phrases can be combined with a comma, for example au.help('baseline,stats').<br />
; [[angularSeparationOfFields]]<br />
: reads an ms and computes the angular separation of all fields, or a subset<br />
; [[buildConfigurationFile]]<br />
: reads the list of antenna stations in an .ms and creates a .cfg file suitable for [http://casa.nrao.edu/docs/taskref/simobserve-task.html simobserve]<br />
; [[editIntents]]<br />
: change the intents for a specified field in an ms (based on John Lightfoot's pipeline script)<br />
; [[getBaselineLengths]]<br />
: returns a dictionary of the baseline lengths in your ms, by default sorted by length<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/GetBaselineStats getBaselineStats]<br />
: returns a bunch of statistics on the baseline lengths in your ms<br />
; [[gjincBeam]]<br />
: compute the effective restoring beam obtained from the casa command {{sdimaging}} when using the GJINC gridding kernel<br />
; [[lstrange]]<br />
: lists the range of LST, UT, MJD for the whole ms, and for scans with OBSERVE_TARGET intent (including the elevation range)<br />
; [[obslist]]<br />
: prints the antenna station coordinates in local offsets from the Center of Array, and computes longest/shortest baselines<br />
; [[planet]]<br />
: contacts the JPL Horizons telnet service and returns the J2000 position, velocity, angular diameter, range and range rate of a solar system object<br />
; [https://safe.nrao.edu/wiki/bin/view/ALMA/PlanetFlux planetFlux]<br />
: calls Bryan Butler's solar_system_setjy module in CASA to compute flux density vs. time or flux density vs. frequency <br />
; [[planetPlots]]<br />
: generates a plots of uv amplitude vs. uv distance for a grid of ALMA configurations and observing frequencies<br />
; [[plotbandpass]] <br />
: a faster version of plotcal for bandpass tables, with useful overlay capabilities. This function has been incorporated into CASA as a task.<br />
; [[plotSunDuringTrack]]<br />
: generates a plot of the Sun's az vs. el during an ms or ASDM<br />
; [[plotconfig]] <br />
: plot any standard telescope configuration of observatories known to casa, and return an array of the sorted baseline lengths<br />
; [[plotmosaic]]<br />
: shows relative location of pointings in an .ms<br />
; [[plotPWV]]<br />
: reads the PWV from each antenna's WVR from the ASDM_CALWVR table in an ms, and creates a plot vs. time<br />
; [[plotweather]]<br />
: plot weather conditions vs. time for your ms<br />
; [[plotWVRSolutions]]<br />
: plot the WVR solutions in terms of baseline-based phase corrections<br />
; [[spectralindex]]<br />
: performs a least-squares fit to the multi-spw output from {{fluxscale}}<br />
; [[timeOnSource]]<br />
: compute the total time spent integrating on-source for each specified field<br />
<br />
<b>A list of additional 1200+ functions, and their brief descriptions, can be found at [https://safe.nrao.edu/wiki/bin/view/Main/CasaExtensions Todd's wikipage]</b></div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=33450ALMA Pipeline Known Issues2022-11-10T14:52:48Z<p>Thunter: /* Known Issues for PL2022.2.0 */</p>
<hr />
<div>== Known Issues for PL2022.2.0 ==<br />
Release date Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|100px]]<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash ([https://open-jira.nrao.edu/browse/PIPE-1468 PIPE-1468]).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved ([https://open-jira.nrao.edu/browse/CAS-13899 CAS-13899]), create a larger image and put the phasecenter a pointing with data. ([https://open-jira.nrao.edu/browse/PIPE-98 PIPE-98])<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. ([https://open-jira.nrao.edu/browse/CAS-13910 CAS-13910])<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR ([https://open-jira.nrao.edu/browse/PIPE-1575 PIPE_1575]).<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation ([https://open-jira.nrao.edu/browse/PIPE-1623 PIPE-1623])<br />
* The new phase offsets QA score in hifa_timegaincal fails ([https://open-jira.nrao.edu/browse/PIPE-1628 PIPE-1628]).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug ([https://open-jira.nrao.edu/browse/CAS-13843 CAS-13843]) in calculating TOPO ranges for such spws ([https://open-jira.nrao.edu/browse/PIPE-1530 PIPE-1530]). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. ([https://open-jira.nrao.edu/browse/PIPE-1631 PIPE-1631]).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal ([https://open-jira.nrao.edu/browse/PIPE-1630 PIPE-1630]).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it rarely means that continuum subtraction needs to be redone (see DRM training material)<br />
<br />
11. Due to CASA bug [https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908], PLWG recommends that all multi-EB ephemeris data be run in serial mode.<br />
<br />
12. The phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
13. In the polcal recipe, if the session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=33449ALMA Pipeline Known Issues2022-11-10T14:49:07Z<p>Thunter: /* CASA */</p>
<hr />
<div>== Known Issues for PL2022.2.0 ==<br />
Release date Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. A bug in imaging ephemeris cubes ([https://open-jira.nrao.edu/browse/CAS-13908 CAS-13908]) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|100px]]<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning (CAS-13899). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved (CAS-13899), create a larger image and put the phasecenter a pointing with data. (PIPE-98)<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. (CAS-13910)<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR (PIPE-1575).<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation (PIPE-1623)<br />
* The new phase offsets QA score in hifa_timegaincal fails (PIPE-1628).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug (CAS-13843) in calculating TOPO ranges for such spws (PIPE-1530). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. (PIPE-1631).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal (PIPE-1630).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it rarely means that continuum subtraction needs to be redone (see DRM training material)<br />
<br />
11. Due to CASA bug CAS-13908, PLWG recommends that all multi-EB ephemeris data be run in serial.<br />
<br />
12. The phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
13. In the polcal recipe, if the session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=33371ALMA Pipeline Known Issues2022-10-27T14:20:07Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Known Issues for PL2022.2.0 ==<br />
Release date Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. a bug in imaging ephemeris cubes (CAS-13908) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|100px]]<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning (CAS-13899). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved (CAS-13899), create a larger image and put the phasecenter a pointing with data. (PIPE-98)<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. (CAS-13910)<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR (PIPE-1575).<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation (PIPE-1623)<br />
* The new phase offsets QA score in hifa_timegaincal fails (PIPE-1628).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug (CAS-13843) in calculating TOPO ranges for such spws (PIPE-1530). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. (PIPE-1631).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal (PIPE-1630).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it rarely means that continuum subtraction needs to be redone (see DRM training material)<br />
<br />
11. Due to CASA bug CAS-13908, PLWG recommends that all multi-EB ephemeris data be run in serial.<br />
<br />
12. The phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
13. In the polcal recipe, if the session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash. The workaround is to set a different reference antenna (or list of reference antennas) in the hif_refant task in the PPR.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=33370ALMA Pipeline Known Issues2022-10-27T14:17:14Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Known Issues for PL2022.2.0 ==<br />
Release date Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. a bug in imaging ephemeris cubes (CAS-13908) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|100px]]<br />
<br />
2. The tclean parameter psfphasecenter is currently not functioning (CAS-13899). If you want to image a mosaic with no data at phase center, then you should make the image larger and move the phase center to a pointing that has data.<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. Until the CASA bug with psfphasecenter is resolved (CAS-13899), create a larger image and put the phasecenter a pointing with data. (PIPE-98)<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. (CAS-13910)<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR (PIPE-1575).<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation (PIPE-1623)<br />
* The new phase offsets QA score in hifa_timegaincal fails (PIPE-1628).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug (CAS-13843) in calculating TOPO ranges for such spws (PIPE-1530). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. (PIPE-1631).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal (PIPE-1630).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it rarely means that continuum subtraction needs to be redone (see DRM training material)<br />
<br />
11. Due to CASA bug CAS-13908, PLWG recommends that all multi-EB ephemeris data be run in serial.<br />
<br />
12. The phase structure function plots in hifa_wvrgcalflag are calculated from antenna-based gains, so baseline-based flags will not be reflected in the plot.<br />
<br />
13. In the polcal recipe, if the session reference antenna is fully flagged on any spw by hifa_gfluxscaleflag, then hifa_gfluxscale will crash.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=32701ALMA Pipeline Known Issues2022-09-24T05:28:07Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Known Issues for PL2022.2.0 ==<br />
Release date Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. a bug in imaging ephemeris cubes (CAS-13908) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|100px]]<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. (CAS-13910)<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR (PIPE-1575).<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation (PIPE-1623)<br />
* The new phase offsets QA score in hifa_timegaincal fails (PIPE-1628).<br />
* hifa_tsyscal or hifa_tsysflag *might* fail due to the recent change to single-pol Tsys (ICT-15219), not communicated to the PL team and for which no E2E9 data were acquired.<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug (CAS-13843) in calculating TOPO ranges for such spws (PIPE-1530). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. (PIPE-1631).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal (PIPE-1630).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it rarely means that continuum subtraction needs to be redone (see DRM training material)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=32660ALMA Pipeline Known Issues2022-09-21T04:04:47Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Known Issues for PL2022.2.0 ==<br />
Release date Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. a bug in imaging ephemeris cubes (CAS-13908) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|100px]]<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. (CAS-13910)<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_flagdata(lowtrans=False) in the PPR (PIPE-1575).<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation (PIPE-1623)<br />
* The new phase offsets QA score in hifa_timegaincal fails (PIPE-1628).<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug (CAS-13843) in calculating TOPO ranges for such spws (PIPE-1530). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. (PIPE-1631).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal (PIPE-1630).<br />
<br />
9. In hifa_spwphaseup, (i) if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0. (ii) if the refant drops out for the majority of solutions the Median Phase RMS assessment cannot be made; (iii) if there is low SNR in the bandpass solutions and the refant jumps in phase then some antennas can incorrectly be identified as outliers by the Median Phase RMS assessment.<br />
<br />
10. For hifa_makeimages(cubes), if mom8fc score is yellow, it rarely means that continuum subtraction needs to be redone (see DRM training material)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=32565ALMA Pipeline Known Issues2022-09-08T21:58:24Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Known Issues for PL2022.2.0 ==<br />
Release date Oct 2022<br />
<br />
=== CASA ===<br />
<br />
1. a bug in imaging ephemeris cubes (CAS-13908) will sometimes cause part of the cube being erroneously empty, clearly visible in the pipeline spectrum under "other QA images": [[File:Europa_sci.spw27.cube.I.iter1.image.spectrum.png|100px]]<br />
<br />
=== Interferometric Pipeline ===<br />
1. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
2. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
3. The atmospheric transmission curves overlaid in h_tsyscal, hifa_tsysflag, and hifa_bandpass will be incorrect if any weather stations have non-sensical pressure data. (CAS-13910)<br />
<br />
4. This version of the pipeline won’t run on Cycle 2 and earlier data that were labeled J2000 frame because astropy does not recognize J2000. The workaround is to set hifa_tsysflag(lowtrans=False) in the PPR (PIPE-1575).<br />
<br />
5. For Single-pol data: (there are none in C8, 2 approved C-rank in C9, so very rare)<br />
* The new calculation of achieved SNR in hifa_spwphaseup will issue a spurious red QA warning that the achieved SNR is zero - the actual calibration and data are fine, this is only a bug in the QA score calculation (PIPE-1623)<br />
* The new phase offsets QA score in hifa_timegaincal fails (PIPE-1628).<br />
<br />
6. Band 5 data with science spws overlapping with the WVR spws might fail in hif_uvcontfit because of a CASA bug (CAS-13843) in calculating TOPO ranges for such spws (PIPE-1530). The workaround is to set a suitable continuum selection range for (only) the failing spw(s) in cont.dat prior to re-running the pipeline.<br />
<br />
7. In hifa_antpos, in the second table "Antenna Position Offsets Sorted By Total Offset", when there are two antennas with the same offset and the Total Offset column has a common value, the second antenna is not bolded in the same way as the first. (PIPE-1631).<br />
<br />
8. In hifa_gfluxscaleflag, the flagging summary table does not reflect flagging applied to the TARGET intent when flags for bad baselines are propagated from PHASE cal (PIPE-1630).<br />
<br />
9. In hifa_spwphaseup, if several antennas of a CM+PM heterogeneous dataset are flagged on the bandpass calibrator, then the value of the Median Phase RMS in the table will be NaN and the QA score will be 0.0.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
<br />
<br />
----<br />
<br />
== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
In addition to the suggestions below, you can start a local http server with python3, either the one included with casa, or the one installed on your system:<br />
<br />
<code>python3 -m http.server 8080 --bind 127.0.0.1 &</code><br />
and then access in your browser the url http://127.0.0.1:8080/<br />
<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
<br />
<br />
----<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset. ''fixed 2022''<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability. ''fixed 2022''<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines. ''fixed 2022''<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator). ''fixed 2022''<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data). ''fixed 2022''<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal. ''fixed 2022''<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels. ''improved 2022, may still require manual examination''<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification. ''fixed 2022''<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656). ''improved 2022''<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432). ''fixed 2022''<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364). ''fixed 2022''<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
<br />
----<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=32471ALMA Pipeline Known Issues2022-07-27T18:05:22Z<p>Thunter: /* Browser security and viewing file://weblog */</p>
<hr />
<div>== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you. (Note that newer versions of firefox (>=v96, ~Feb 2022) no longer have the parameter "privacy.file_unique_origin", but in that case the same workaround described below can be achieved by setting "security.fileuri.strict_origin_policy" to False).<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset.<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability.<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines.<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator).<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data).<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal.<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification.<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656).<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
13. An irregular mosaic with no pointing at the geometric center will likely fail, and the pipeline will warn you of this likelihood. You can use tclean with the psfphasecenter parameter to image manually. (PIPE-98)<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=32218ALMA Pipeline Known Issues2022-03-31T12:24:21Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you:<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset.<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability.<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines.<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator).<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data).<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal.<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification.<br />
<br />
9. MOUS with large numbers of EBs/spws/fields/integrations can exceed the 100MB memory footprint of bsend in CASA and crash (PIPE-1337). This feature was introduced in 2014 and a small increase has been requested (CAS-13656).<br />
<br />
10. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
11. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
12. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=32217ALMA Pipeline Known Issues2022-03-31T12:20:28Z<p>Thunter: /* Known Issues for PL2021.2.0 */</p>
<hr />
<div>== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you:<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset.<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability.<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines.<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator).<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data).<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal.<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
1. hsd_imaging does not properly combine datasets associated with virtual spw. (PIPE-1478)<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=32147ALMA Pipeline Known Issues2022-03-28T17:03:29Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you:<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset.<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability.<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines.<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator).<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data).<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal.<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. Full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
16. If the check source is so faint that no gain solutions can be found, then hifa_gfluxscale will crash (PIPE-1468).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=31326ALMA Pipeline Known Issues2021-11-09T17:25:21Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you:<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset.<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability.<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines.<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator).<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data).<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal.<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
15. If pointing is not performed in every EB of a full-polarization dataset, hif_makeimages (check sources) can crash (PIPE-1364).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunterhttps://casaguides.nrao.edu/index.php?title=ALMA_Pipeline_Known_Issues&diff=31321ALMA Pipeline Known Issues2021-11-08T20:55:32Z<p>Thunter: /* Interferometric Pipeline */</p>
<hr />
<div>== Browser security and viewing file://weblog ==<br />
More stringent security in most web browsers in 2020 broke the weblog when displayed using a file:// URL, and users may encounter "failure to load" errors ([Firefox security update details|https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp]). As of PL2020, the weblog contains instructions for what users can do if they encounter this error, including a new local html server embedded in the pipeline that users can access through CASA. These are applicable for the older weblogs, even if the weblog itself won't give them to you:<br />
<br />
[[File:Weblog.security.png]]<br />
<br />
== Known Issues for PL2021.2.0 ==<br />
Release date Oct 2021<br />
<br />
=== CASA ===<br />
<br />
1. The bsend mechanism has a limit of 100 MB (introduced in 2014), which can cause the pipeline to fail (PIPE-1337) on MOUS with large numbers of EBs/spws/targets/integrations. So far, it has been seen only on a 42-EB 7m dataset.<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
1. CASA tclean sometimes exits with an unsupported code “0” - this generally is okay, but PL will capture that as a yellow warning in the weblog, and the data reducer should check the images for scientific acceptability.<br />
<br />
2. In the hifa_almarenorm PDF summary plot, if the first spw is narrow, subsequent spws even if wide will be missing the vertical segment boundary lines. This only affects the summary plot, so other plots in the PDF will show the lines.<br />
<br />
3. When spw mapping is in use, bad data can sometimes escape unassessed by hifa_bandpassflag and appear as bad in hifa_timegaincal and hif_applycal (PIPE-838, to be fixed by PIPE-1154 where spw mapping and combination will be prohibited for all calibrators except the phase calibrator).<br />
<br />
4. If an spw is fully flagged for Tsys in hifa_tsysflag, then hifa_gfluxscale can crash (PIPE-1208). This is often due to poor or strongly-varying atmospheric transmission vs. frequency. The possible workarounds are to either: set the offending EB to SemiPass, raise ff_max_limit from 13 to 30 for hifa_tsysflag in the PPR, or flag the poorest transmission parts of the spw (both its Tsys solutions and visibility data).<br />
<br />
5. For the hif_applycal stage, the QA subscore for %newly flagged is not included in the overall score evaluation, meaning that a “!” or “?” can appear in the left navigation pane, different from the “By Topic” QA color bar for hif_applycal.<br />
<br />
6. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
7. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
8. <span style="color: red"> full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be removed from that task</span> in casa_pipescript.py in order for casa_pipescript.py to run (PIPE-1226). casa_restorescript.py works without modification.<br />
<br />
=== Single Dish Pipeline ===<br />
<br />
0. Some task parameters in PPR are not selected properly. For example, hsd_blflag fails when spw is specified in PPR (PIPE-1298).<br />
<br />
== Known Issues for PL2020.1.0 ==<br />
Release date Feb 2021<br />
<br />
=== CASA ===<br />
[https://casa.nrao.edu/casadocs/casa-6.1.0/introduction/known-issues CASA 6.1.X known issues]<br />
<br />
=== Interferometric Pipeline ===<br />
<br />
0. For polarization datasets, hifa_restoredata does not include the polarization calibrator in its applycal commands. A workaround is being arranged with DRM.<br />
<br />
1. If hifa_lowgainflag flags a large number of antennas on a given spw, but not all antennas, then the imaging pipeline can crash downstream in hif_findcont. One workaround is to set niter=2 in hifa_lowgainflag in the PPR. This works as desired (the whole spw is flagged), but it fails to render the weblog page for hifa_lowgainflag. The other alternative is to apply a manual flag for the spw in the regular flag template file.<br />
<br />
2. A bad score for the mom8fc can be erroneously triggered in cube hifa_makeimages when the channelized noise is not uniform in the findcont channels, and has some channels with significantly higher noise (e.g. in regions of deep atmospheric absorption), resulting in more pixels above the Peak/(medianChanMAD) than expected. No workaround - looking at weblog plots should make it clear that there is really no emission in the findcont channels.<br />
<br />
3. For full-polarization datasets, the amp vs. time detail plots (per antenna) in hif_applycal do not include the calibrator scans with POLARIZATION intent, but the plots of calibration slutions in hifa_timegaincal do include them. The workaround is to run plotms manually if there is bad data in the parent plot (with all antennas overlaid) and you want to determine the subset of antennas involved.<br />
<br />
4. The amplitude vs uv distance version of the “before/after flagging” plots are not shown in hifa_polcalflag. Only the ampl. vs time plots are shown.<br />
<br />
5. Datasets suffering from the Cycle 7 correlator instrumental problem related to concurrent subarrays can have some outliers escape unflagged. But this issue was apparently fixed in SCCB-1042 (2021-05-20), so it should not affect the remainder of Cycle 7 data.<br />
<br />
6. Antenna-based single-polarization, single-integration outliers not apparent in hifa_polcalflag can appear as amplitude outliers on the polarization calibrator in hif_applycal.<br />
<br />
7. When spw mapping is used in hifa_spwphaseup, bad data on the bandpass calibrator can escape hifa_bandpassflag unflagged and appear as outliers in hif_applycal, because the timegaincal solutions come from a different spw when applycal is run. <br />
<br />
8. The QA message in hifa_gfluxscale on the ratio of S_calibrated / S_catalog has an incorrectly calculated percentage.<br />
<br />
9. For Cycle 3 datasets affected by extra field IDs associated with only SQLD data, the new hifa_targetflag stage will show a warning message for these fields (each spw): "Warning! Unable to compute flagging for intent TARGET, field 2, spw 16". The workaround is simply to ignore this warning – it should not happen in new data. <br />
<br />
10. Due to CASA changing how tasks are built, several tasks fail to completely render the "inp" input parameters (blank lines appear in the "inp" output). help(taskname) still lists all of the available parameters, and we strongly recommend users refer to the Reference Manual if task parameters are not clear. Affected tasks: hifa_importdata, hifa_wvrgcalflag, hif_lowgainflag, hifa_bandpassflag, hifa_bandpass, hif_applycal, hif_makeimlist, hif_makeimages, hifa_imageprecheck, hif_checkproductsize, hifa_exportdata, hif_mstransform, hif_findcont, hsd_importdata, hsd_flagdata, hsd_applycal, hsd_imaging,<br />
hifa_polcalflag, hif_uvcontfit, hif_uvcontsub (dryrun parameter only missing)<br />
<br />
11. In the Cycle 6, Cycle 7, and PL2020 releases: datasets with heterogenous antenna diameters will also include the cross-diameter baselines in the science target imaging (rather than merely the 7m-7m or merely the 12m-12m baselines). The calibrator imaging stage correctly includes all baselines, as desired. For 7m array projects with 1 or more 12m antennas included, the effects on the resulting image are: a different effective PB response (slightly smaller), lower noise, a smaller synthesized beam, and (likely) a smaller LAS. Different science use cases may see the net difference as a positive or negative effect. <br />
<br />
12. In all pipeline releases through PL2020: FDM spws with 256 channels are being labeled as TDM on the Spectral Setup details page. This is due to faulty logic implemented in CASA::msmd.almaspws.<br />
<br />
13. full-polarization datasets contain a hifa_lock_refant() task - 'pipelinemode="automatic"' must be remove from that task in casa_pipescript.py in order for casa_pipescript.py to run. casa_restorescript.py works without modification.<br />
<br />
14. As in previous releases, in the hif_applycal weblog, if there are multiple phase calibrators or check sources, their calibrated spectra are overlaid in the same plot (PIPE-432).<br />
<br />
=== Single Dish Pipeline ===<br />
1. Observed RMS tends to be larger than theoretical RMS (SD pipeline). <br />
There are two possible causes. One is that ATM and ripple features remain in the final image. The second is that the pixels located at the map edge are included in the measurement of the observed RMS. The latter will be improved in pipeline2021.<br />
<br />
2. Strong emission is flagged in hsd_blflag. <br />
The reason of this issue is that some emission components at some channels (i.e. wing component) are not identified as a line. This increases the RMS at corresponding channels. Essentially, improving the line identification algorithm is required to fix this issue. This issue can be avoided by changing the threshold of blflag manually. <br />
<br />
3. Wrong position of the open circle indicating peak SN position in Peak SN map. <br />
When the scan direction is not along RA/DEC, the open circle indicating peak SN position in Peak SN map and spectrum at peak position are not plotted correctly. However, the information of the peak position is used just as a reference. This does not affect the detection of the contamination and giving “warning”.<br />
<br />
4. hsdn_restoredata crashes (Nobeyama Pipeline)<br />
<br />
5. executeppr does not support hsd_restoredata task (ALMA-SD Pipeline)</div>Thunter