User:Taniguchi
Cycle Compatibility and New Tool
From Cycle 11 (Pipeline 2024), hsd_imaging calls tsdimaging, not the former sdimaging.
About This Guide
Most recently updated for CASA Version 6.6.1 using Python 3.8
This guide describes some examples for creating and perfecting the Total Power (TP) imaging products using the ALMA Cycle 11 Pipeline, for pipeline & manually calibrated data.
If your data were manually imaged by ALMA, you should instead consult the scriptForImaging.py delivered with your data.
The Section Restore Pipeline Calibration and Prepare for Re-imaging describes the first steps to do. After that, the individual sections are self-contained (and they typically assume the "Restore" has been performed). It illustrates how to completely re-run the pipeline from beginning to end in order to reproduce the pipeline run done at your ARC.
Additional documentation on the Cycle 11 pipeline can be found in the Pipeline User's Guide which can also be found at the ALMA Science Portal. The User's guide describes how to obtain the ALMA Pipeline, how to use it to calibrate and image ALMA interferometric (IF) and single-dish (SD) data, and a description of the Pipeline WebLog.
Note that the scripts described in this guide have only been tested in Linux (RedHat 8) and Python 3.8. Before CASA 5.6.x, the pipeline scripts are not written in Python 3 and may not work properly.
As written in the Pipeline User's Guide, CASA task sdbaseline is recommended for manual baseline subtraction.
Getting and Starting CASA
If you do not already have CASA installed on your machine, you will have to download and install it.
Download and installation instructions are available here:
http://casa.nrao.edu/casa_obtaining.shtml
In this guide, we process ALMA Cycle 8 data with CASA 6.6.1.17 and pipeline-2024.1.0.8.
To use pipeline tasks, you must start CASA with
casa --pipeline
Restore Calibration and Prepare for Re-imaging
STEP 1: Follow instructions in your QA2 report for restoring pipeline calibrated data using the *scriptForPI.py. In general, scriptForPI.py is only compatible with CASA versions similar to the one used for its creation. See the Table at https://almascience.org/processing/science-pipeline for details. For running *scriptForPI.py, you move to the "script" folder containing script files and run the script using the following command (modify <uid_name> accordingly):
execfile('member.<uid_name>.scriptForPI.py')
If you want to obtain the data sets that are the same as the products without any changes, just run scriptForPI.py. scriptForPI.py stops at the calibration stage without atmospheric correction and baseline subtraction, if the script folder contains member.<uid_name>.casa_piperestorescript.py. Follow the procedures in ALMA Pipeline User's Guide to obtain the final datasets with atmospheric correction and baseline subtraction.
If the script folder does not contain member.<uid_name>.casa_piperestorescript.py, but contains member.<uid_name>.casa_pipescript.py, scriptForPI.py uses the latter python script. In that case, all of the stages included in casa_pipescript.py are running including atmospheric model correction and baseline subtraction. Running scriptForPI.py without casa_piperestorescript.py enables us to obtain all of the intermit files (<uid_name>.ms, <uid_name>.ms.atmcor.atmtypeX, and <uid_name>.ms.atmcor.atmtype1_bl, please see ALMA Pipeline User's Guide for suffix).
Once completed, the following files and directories will be present. More information on the structure in the ALMA Archival Data Primer
- calibrated/
- This directory contains a file(s) called <uid_name>.ms.split.cal (one for each execution in the MOUS), products/, working/, and rawdata/ subdirectories. In the working/ subdirectory, <uid_name>.ms are restored. For further data processing, see ALMA Pipeline User's Guide.
- If you run the script without casa_piperestorescript.py, only the working subdirectory is created and all data is stored in this subdirectory.
- calibration/
- This directory contains auxproducts.tgz, auxcaltables.tgz, caltables.tgz, auxproducts.tgz, and flagversions.tgz. The text files (.txt) include information of the applied commands.
- product/
- This directory contains the original image products (fits format).
- qa/
- This directory contains the original weblog and the QA2 and QA0 reports. The QAs reports contain summaries of the scheduling block (SB), and calibration and imaging results.
- If you want to obtain the same results with the QA2 report, you should check the CASA version in the reports and run the scriptForPI.py script with the same version.
- raw/
- This directory contains the raw asdm.sdm(s).
- script/
- This directory contains the file scriptForPI.py (named member.<uid_name>.scriptForPI.py) which internally runs member.<uid_name>.hsd_calimage.casa_piperestorescript.py and other necessary tasks to restore the data.
- The folder also contains member.<uid_name>.hsd_calimage.casa_pipescript.py, a full CASA pipeline script that reproduces all pipeline products.
Here, we run the scriptForPI.py without casa_piperestorescript.py to obtain the final images and evaluate the pipeline procedures.
You can find the results of each pipeline task in WebLog.
STEP 2: Copy member.<uid_name>.casa_pipesescript.py in "calibrated/working/" directory. Change to a directory that contains the calibrated data suitable for running pipeline imaging tasks (i.e. *.ms) called "calibrated/working" after the pipeline restore.
Change applied atmospheric model (hsd_atmcor stage)
You can find which atmospheric model is applied from the ms name: <uid>.ms.atmcor.atmtypeX, where X (1-4) indicates the applied model; atmType=1 (tropical), 2 (mid-latitude summer), 3 (mid-latitude winter), and 4 (subarctic summer). Details are summarized in Sawada et al. (2021). The pipeline applies the most optimal one for each EB. You can check the other models in the ATM Heurestics Plots.
If you want to change the applied atmospheric model, you can select them by giving either a single integer (apply to all EBs) or a list of integers (models per EB) to atmtype parameter. Furthermore, you can constrain parameters such as antenna and spectral window. See more details in the Pipeline Reference Manual.
For instance, you can select a particular data set(s), spw, and applied atmospheric model. You can edit hsd_atmcor command in pipescript.py as bellow:
hsd_atmcor(infiles = ['uid___A002_X10bf6e3_Xe1a.ms', 'uid___A002_Xfc10af_X33f1.ms'], spw = '25', atmtype=2)
Change baseline subtraction (hsd_baseline stage)
At hsd_baseline stage in WebLog, you can find the applied baseline as red lines as well as before and after baseline subtraction spectra (blue lines). The hsd_baseline and hsd_blflag stages run twice. Pipeline conducted baseline fitting using a cubic spline function. Broad lines are sometimes subtracted as baselines. The attached figure is an example of this case (spw 21).
For this spw 21, the base baseline is likely the 1st-order polynomial baseline. If you want to change the baseline
hsd_baseline(infiles = vis, linewindow={21:[500,1900]}, fitfunc='poly', fitorder = 1)
hsd_blflag()
hsd_imaging()
You edit the hsd_baseline command in pipescript.py which is restored in the script folder as follows:
hsd_baseline(linewindow={21:[500,1900]}, fitfunc='poly', fitorder = 1)
The hsd_baseline stage runs twice and then you put this command for two parts. And then, rerunning scriptForPI.py in casa.
The simplest case is to start initially.
execfile('member.<uid_name>.scriptForPI.py')
Then, you will obtain the WebLog. The linewindow parameter is set at channel range(s) where the line(s) are detected. Regarding how to set the parameters, see the Pipeline Reference Manual. The attached figure shows the result of the manual baseline fitting.
NOTE: The current single-dish pipeline applies the same fit function to all of the spectral windows. Thus, if you change fintfinc = 'poly', this is applied to all of the spectral windows even if the cubic spline function works in the original results. The individual fitting function is developed in the future pipeline version.
